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Maritime  Domain  Awareness:  Assessment  of  Current  Status 


Abstract 

Maritime  Domain  Awareness  (MDA)  is  viewed  as  predominately  a  security  issue,  however  the 
economic  ramifications  of  an  attack  against  a  high-value  target  such  as  a  container  vessel,  cruise 
ship,  or  petro-chemical  facility  elevate  the  problem  from  a  national  level  to  cause  for  global 
concern  (Allen,  2008).  A  significant  attack  could  cause  the  port  to  shut  down  and  spread  anxiety 
throughout  the  global  marketplace.  To  mitigate  the  danger  posed  by  maritime  vessels,  the  United 
States  (US)  Navy,  Coast  Guard,  and  law  enforcement  agencies  need  greater  maritime  domain 
awareness,  appropriate  legal  agreements,  and  partnerships.  A  large  effort  is  underway  to  develop 
technologies  to  help  maintain  MDA,  legal  arrangements  for  sharing  infonnation,  and  guidance 
for  military  and  law  enforcement  personnel  on  procedures  to  deal  with  this  multi-faceted 
problem.  In  this  paper,  we  report  on  the  progress  made  in  2008  to  improve  the  effectiveness  of 
systems  provided  to  the  warfighter  in  MDA  missions  (including  Coast  Guard,  law  enforcement, 
and  other  agencies).  In  line  with  the  theme  for  this  year’s  symposium  -  command  and  control 
(C2)  and  agility  -  the  maritime  threat  environment  is  a  complex  problem  with  multiple  variables 
and  requires  a  number  of  agile  solutions. 

Introduction 

Maritime  Domain  Awareness  (MDA)  is  a  National  Security  concept  that  relies  on  the  aggregate 
capabilities  of  multiple  government  agencies  such  as  the  Department  of  Defense  (DoD)  and 
Department  of  Homeland  Security,  as  well  as  other  federal,  state,  and  local  agencies  to  achieve 
comprehensive  situational  awareness  of  any  threat  associated  with  the  maritime  domain.  The 
National  Plan  to  Achieve  Maritime  Domain  Awareness  (Department  of  the  Navy,  2005)  defines 
the  maritime  domain  as  “all  areas  and  things  on,  under,  relating  to,  adjacent  to,  or  bordering  on  a 
sea,  ocean,  or  other  navigable  waterway,  including  all  maritime-related  activities,  infrastructure, 
people,  cargo,  and  vessels  and  other  conveyances.”  Moreover,  the  National  Plan  identifies 
nation-state,  terrorist,  transnational  criminal  and  piracy,  and  environmental  and  social  threats 
within  the  maritime  domain.  In  order  to  address  these  threats,  the  National  Plan  requires  the 
capability  to:  (a)  persistently  monitor,  in  the  global  maritime  domain,  vessels  and  craft,  cargo, 
vessel  crews  and  passengers,  in  all  identified  maritime  situation  awareness  areas  of  interest;  (b) 
access  and  maintain  data  on  vessels,  facilities,  and  infrastructure;  (c)  collect,  fuse,  analyze,  and 
disseminate  infonnation  to  decision  makers  to  facilitate  effective  understanding;  and,  (d)  access, 
develop  and  maintain  data  on  MDA-related  mission  perfonnance. 

The  DoD,  following  guidance  set  forth  in  the  National  Concept  of  Operations  for  MDA, 
developed  the  Fleet  Concept  of  Operations  for  Maritime  Domain  Awareness  (Department  of  the 
Navy,  2009)  and  the  Navy  MDA  Concept  (Department  of  the  Navy,  2007),  which  describe  the 
Fleet  role  in  MDA  and  how  Fleet  commanders  will  develop  and  maintain  MDA  to  accomplish 
Navy  missions  across  the  full  Range  of  Military  Operations.  These  and  other  publications 
provide  a  foundation  for  developing  interagency  and  agency-specific  policies,  processes, 
procedures,  and  organizational  relationships  to  align  activities  that  contribute  to  achieving  MDA 
throughout  the  Global  Maritime  Community  of  Interest.  The  U.S.  Navy  has  the  responsibility  to 
develop  a  comprehensive,  worldwide  MDA  capability.  The  Deputy  Under  Secretary  of  the  Navy 
has  appointed  the  Naval  Postgraduate  School  (NPS)  to  assess  the  overall  capabilities  and  needs 
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of  MDA.  In  this  paper  we  report  on  the  progress  made  in  2008  to  improve  the  effectiveness  of 
the  warfighter  (including  Coast  Guard,  law  enforcement,  and  other  agencies)  in  MDA  missions. 

This  effort  focused  primarily  on  the  set  of  MDA  systems  that  the  Program  Executive  Office 
(PEO),  Command,  Control,  Computers,  Communications,  and  Intelligence  (C4I)  has  designated 
Spiral- 1.  Our  assessment  is  based  on  data  from  numerous  tests,  experiments,  and  studies  of 
MDA  systems  directed  by  PEO  C4I  and  conducted  by  Operational  Test  &  Evaluation  Force 
(OPTEVFOR)  or  by  NPS.  Specifically,  we  have  aggregated  and  analyzed  data  from 
observations,  surveys,  chat  logs,  and  interviews  from  six  experimental  and  empirical  exercises 
where  operators  used  MDA  Spiral- 1  technologies,  and  from  other  events  in  which  those  technol¬ 
ogies  were  discussed  by  operational  personnel.  We  applied  a  comprehensive  assessment 
structure  and  methodology  (Schacher,  &  Freeman,  2008)  that  was  developed  and  validated  in  the 
Trident  Warrior  series  of  exercises  that  began  in  2002.  The  assessment  structure  was  customized 
to  MDA,  and  it  can  be  used  throughout  the  MDA  program  to  evaluate  technologies, 
organizations,  processes  and  other  enabling  components  of  the  MDA  solution.  Our  findings  and 
recommendations  are  presented  using  this  structure. 

Background 

This  objective  for  this  NPS  effort  was  to:  (1)  Refine  a  project  plan  that  provides  a  concept  of 
operations  (CONOP)  around  the  core  operational  threads  (e.g.,  standard  work  flows,  or  “business 
practices”)  to  be  then  further  used  for  operational  field  experimentation  in  Trident  Warrior  08; 

(2)  Specify  measures  and  metrics  related  to  decision  making  and  the  continued  evolution  of 
MDA  system  elements  that  will  impact  the  Global  War  On  Terror,  and  are  also  consistent  with 
the  DoD,  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  experimentation  and 
acquisition  program  needs;  (3)  Integrate  efforts  across  MDA  working  groups,  brought  together  in 
operational  testing  venues,  under  a  consistent  experiment  design  process  that  will  also  include 
standard  metrics  developed  for  MDA  analysis  of  capabilities  to  ultimately  assess  Spiral- 1 
capabilities  against  fleet  requirements. 

Accordingly,  this  effort:  (1)  defined  the  current  MDA  workflow  that  new  systems  must  support 
and  how  these  new  technologies  will  impact  the  current  MDA  workflow;  (2)  defines  a  measure¬ 
ment  and  assessment  framework  for  evaluating  MDA  systems  (technological,  procedural,  and 
organizational);  and,  (3)  applies  this  framework  in  an  assessment  of  MDA  systems. 

In  2007  the  Secretary  of  the  Navy  directed  the  fielding  of  a  prototype  MDA  capability  by  August 
2008,  and  established  a  Cross  Functional  Team  (CFT)  to  oversee  the  effort.  Spiral- 1  will:  (a) 
Provide  a  capability  to  the  U.S.  Central  Command  (CENTCOM)  and  U.S.  Pacific  Command 
(PACOM)  areas  of  responsibility  (AORs),  interagency  partners,  and  select  friendly  and  allied 
nations,  (b)  Create  a  network  that,  at  multiple  levels  of  security  and  across  multiple  domains,  will 
feed  many  data  streams  into  a  common  operational  picture  (COP)  accessible  throughout  the 
United  States  Government  and  foreign  or  Coalition  partners,  (c)  Be  able  to  handle  time-sensitive 
maritime  threats,  (d)  Be  designed  for  expansion.  The  effort  will  be  used  to  resolve  or  develop 
new  policy  and  procedures  for  MDA.  Subsequent  spirals  will  extend  this  capability  and  add 
functionality. 

MDA  Spiral-1  Technologies 

This  assessment  reported  on  in  this  paper  focused  predominately  on  the  eight  Spiral- 1 
technologies,  which  were  tested  in  2008.  Several  other  technologies  deemed  to  have  high 
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potential  impact  on  the  MDA  mission  were  also  included  in  this  assessment.  For  a  more  detailed 
description  of  the  tools  see  Hutchins,  Gallup,  MacKinnon,  Schacher,  Miller,  Freeman,  Dunaway 
&  Poeltler,  2008.  These  technologies  are: 

•  Comprehensive  Maritime  Awareness  (CMA)  -  The  Naval  Research  Laboratory’s 
enhanced  vessel  tracking  project 

•  Law  Enforcement  Information  Exchange  (LInX)  -  The  Naval  Criminal  Investigative 
Service’s  non-classified  system  for  information  sharing  of  law  enforcement  information 

•  Global  Trader  -  The  Office  of  Naval  Intelligence’s  cargo  data  and  anomaly  detection  tool 

•  Maritime  Global  Network  (MAGNet)  -  The  Coast  Guard’s  intelligence  program 
backbone  database  with  enhanced  anomaly  detection  for  people 

•  Tactical  Electronic-Maritime  Interdiction  Operation  (E-MIO)  System  Wireless  -  An 
E-MIO  data  collection  and  transfer  system 

•  Fast  Connectivity  for  Coalition  Agents  Program  (FAST2CAP)  -  A  common  maritime 
operating  picture  that  allows  the  watchstander  to  construct,  implement  and  reconfigure 
search  agents 

•  Tripwire  -  Office  of  Naval  Intelligence’s  threat  detection  tools 

•  Google  Earth  -  A  commercial  toolset  for  fusing  data  and  displaying  it  on  a  globe 

Data  concerning  other  non-Spiral-1  MDA  systems  (e.g.,  MDA  Data  Sharing  Community  of 
Interest  (DS  COI),  a  system  for  disseminating  automatic  identification  system  (AIS)  data,  and 
Predictive  Analysis  for  Naval  Deployment  Activity  (PANDA)  were  also  collected  during  the  test 
events  and  these  results  are  also  included  in  this  report. 

In  the  following  sections  we  describe  our  method  of  analysis,  summarize  our  findings,  present 
recommendations,  and  present  detailed  findings  in  each  assessment  area. 

Assessment  Method 

The  assessment  method  used  for  this  MDA  effort  extends  the  methodology  developed  and 
refined  for  the  Trident  Warrior  series  of  exercises,  which  has  continued  to  be  refined  since  2002. 
The  elements  of  this  method  include:  (1)  Define  a  framework  for  assessment  in  the  MDA 
domain;  (2)  Collect  data  from  test  and  experimental  venues;  and  (3)  Categorize  data  using  the 
framework,  assign  scores  to  categorized  data,  and  summarize  the  data.  These  steps  are  described 
in  more  detail  below. 

Assessment  Framework 

The  MDA  assessment  framework  specifies  a  three-level  hierarchy  of  assessment  areas,  attributes 
that  apply  to  each  assessment  area  and  the  associated  metrics  used  to  evaluate  perfonnance  for 
that  attribute,  and  specific  measures.  For  example,  the  attribute  ‘accessible’  -  when  applied  to 
measuring  the  sub-area  Information  Retrieval,  under  Technical  Performance  -  would  be 
measured  by  collecting  data  on  both  the  percentage  of  time  information  is  available,  and  how 
efficient  the  process  was  for  retrieving  infonnation,  i.e.,  number  of  steps  involved  to  access  the 
information.  However,  when  applying  the  attribute  ‘accessible’  to  the  sub-area  Information 
Sharing  (also  under  Technical  Perfonnance)  it  would  include  a  combination  of  a  measure  for 
compatible  (system  interoperability),  in  addition  to  measures  for  available,  and  efficient.  This 
framework  is  designed  to  address  technological  systems,  organizational  structures  and  processes, 
policy  or  guidance,  and  other  factors  that  ensure  a  robust  mission  capability.  (See  MDA 
Assessment  Areas  and  Structure,  in  Appendix  A.) 
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The  assessment  framework  addresses  five  distinct  assessment  areas,  and  two  levels  of  sub-areas 
within  them.  The  live  assessment  areas  are: 


•  System  Performance  concerns  how  well  a  system  performs  its  functions,  its  support  of 
MDA  operations,  warfighter  acceptance,  automation,  and  system  management  and 
security  functions. 

•  Operations  Performance  addresses  the  quality  of  knowledge  management  concerning 
vessels  of  interest  (Vols),  MDA  intelligence,  surveillance,  and  reconnaissance  (ISR),  and 
MIO. 

•  Warfighter  Performance  focuses  on  operator  acceptance  and  understanding  of  the  MDA 
mission,  as  well  as  unit  and  individual  capability  to  execute  that  mission. 

•  Organization/Guidance  focuses  on  the  fit  of  organizational  structures  and  processes  to 
the  MDA  mission  including  Maritime  Headquarters  with  Maritime  Operations  Center 
(MOC),  the  sufficiency  of  agreements  between  entities,  and  the  adequacy  of  guidance 
within  entities. 

•  Supportability  and  Readiness  concerns  factors  that  ensure  MDA  Spiral- 1  systems  are 
robust  and  reliable. 

Capability  indicators  (or  assessment  metrics)  have  been  defined  for  each  of  the  Assessment 
Areas.  These  indicators  include  both  attributes  and  their  measures.  (These  capability  indicators 
were  taken  from  those  used  for  the  Naval  Network  Warfare  Command  Capabilities  Based 
Analyses  for  network-centric  operations  (NCO),  command  and  control  (C2),  and  battle  space 
awareness.  Table  1  shows  the  attributes  used  for  the  MDA  assessment,  however  not  all  are  used 
for  each  Assessment  Area.  The  principal  indicators  are  Effectiveness  and  Military  Utility. 

Table  1.  MDA  Capability  Indicators  (or  Assessment  Metrics). 


Area 

Effective 

Utility 

Area 

Effective 

Systems 

Accessible 

Capable 

Reliable 

Usable 

Improved 

Needed 

Applicable 

Wanted 

Warfighter 

Capable 

Reliable 

Operations 

Accessible 

Capable 

Reliable 

Organization 

/Guidance 

Accessible 

Capable 

Usable 

Specific  measures  and  the  type/s  of  data  required  to  are  defined  for  each  of  the  capability 
indicators.  Examples  are  shown  in  Table  2  and  the  full  list  is  contained  in  Appendix  B.  Metric 
Attributes  are  shown  in  bold  and  their  measures  in  plain  text.  Use  of  the  structure  presented  in 
Appendix  A  provides  a  consistent  approach  to  MDA  assessment  and  reporting  across  test  venues 
that  were  used  during  2008.  This  structure  was  used  to  correlate  results  from  a  variety  of  sources. 
Definitions  of  the  attributes  used  in  this  assessment  framework  are  found  in  Appendix  C. 
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Table  2.  Example  of  MDA  Assessment  Metrics:  Assessment  Area,  Capability  Indicators, 


and  Metrics.  (See  Appendix  B  for  complete  set.) 

Measure  of  Effectiveness 

System  Performance 
Assessment  Area 

Capability  Metrics 

Indicators 

Technical 

Performance 

Improved:  -5  to  +5  rating  of  improvement  over  existing  systems ,  by  system 

aspect. 

Needed:  system  fills  a  gap  in  existing  capabilities,  Y/N. 

Applicable:  system  is  applicable  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  available. 

Information 

Retrieval 


Accessible:  roll-up  of  information  accessibility. 

Available:  %  of  time  information  is  available. 

Efficient:  number  of  steps  to  access  information. 


Capable:  roll-up  of  capability  to  retrieve  required  information. 
Sufficient:  %  of  information  needed  for  assessment. 

Timely:  time  required  to  retrieve  information. 


Data  Collection 

Data  were  collected  during  multiple  site  visits  and  several  Spiral- 1  system  test  events. 

1 .  As  part  of  the  Workflow  Analysis  and  Process  Engineering  Workshop  (PEW)  an 
overview  of  MDA  activities  was  conducted  during  site  visits  to  US  Naval  Forces  Central 
Command  (NAVCENT),  ONI,  Second  Fleet,  Third  Fleet,  Fifth  Fleet,  and  Sixth  Fleet. 

The  objective  was  to  document  the  way  MDA  is  currently  done  and  to  identify  where 
new  technologies  are  expected  to  have  the  greatest  impact.  The  workflow  for  MDA  as 
currently  executed  was  documented  in  these  visits  (Freeman,  Gallup,  MacKinnon,  and 
Hutchins,  (2008).  The  workflow  was  validated  in  the  PEW  held  at  the  Naval 
Postgraduate  School,  Monterey,  CA,  15-17  Jan  2008,  and  was  mapped  to  the  Maritime 
Headquarters  with  Maritime  Operations  Center  (MHQ  w/MOC)  core  processes  in  a 
workshop  in  Norfolk,  VA,  in  29  January  2008. 

2.  The  Technical  Risk  Reduction  Fimited  Objective  Experiment  was  held  at  Space  and 
Naval  Systems  Center  (SPAWAR),  Fab  140,  San  Diego,  CA,  2-6  June  2008,  and  was  the 
first  simultaneous  testing  of  multiple  MDA  technologies  using  Fleet  participants.  The 
objective  was  to  go  through  the  test  procedure  to  identify  potential  problems,  with 
technologies,  methodology,  scenario,  and  so  on,  prior  to  the  primary  test  event. 

3.  The  Visit  Board  Search  and  Seizure  (VBSS)  School,  in  San  Diego,  CA,  was  visited  19 
June  2008  to  observe  E-MIO  usage  and  effectiveness. 

4.  Trident  Warrior  08  was  held  aboard  multiple  Navy  ships  to  test  approximately  100  newly 
developed  technologies  and  policies,  15  Jun  -15  July  2008. 

5.  Empire  Challenge  08  was  held  at  China  Fake  Naval  Station  and  in  San  Francisco  Harbor 
as  well  as  multiple  other  sites,  7-3 1  July  2008. 

6.  FAIRGAME  was  held  at  simultaneous,  multiple  sites:  Naval  Maritime  Intelligence 
Center  (NMIC),  NAVCENT,  US  Pacific  Fleet  Command  (PACFFT),  Maritime 
Intelligence  Fusion  Center,  Pacific  (MIFCPAC),  MIFCFANT,  SPAWAR  Systems 
Center,  San  Diego  (SSC-SD),  and  Naval  Criminal  Investigation  NCIS),  15-18  July  2008, 
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The  data  gathered  in  these  venues  were  largely  qualitative  -  including  survey  comments,  chat 
logs,  data  collectors’  observations,  and  interview  notes  -  though  some  quantitative  survey  data 
were  also  collected.  As  noted  above,  the  data  largely  concerned  the  Spiral- 1  systems  evaluated  in 
most  of  these  venues  however,  data  were  also  collected  on  other  systems  with  potential  MDA 
utility.  In  sum,  the  systems  assessed  were:  CMA,  E-MIO  Wireless,  FASTC2AP,  Global  Trader, 
Google  Earth,  LiNX,  MAGNET,  MASTER,  MDA  DS  COI,  MIDAS,  PANDA,  and  Tripwire. 

Analysis  Method 

Data  were  aggregated  across  all  venues  and  each  datum  was  assigned  to  one  or  more  areas  (e.g., 
System  Perfonnance)  of  the  assessment  framework  (depicted  in  Appendix  A)  at  the  lowest  level 
of  detail  (e.g.,  Technical  Performance,  or  sub-sub-areas).  Each  item  in  each  category  was  rated  to 
indicate  that  it  described  a  strength  (score  =  3),  a  concern  (score  =  2),  or  a  deficiency  (score  =  1) 
in  an  MDA  system,  suite  of  systems,  or  deployment  concept  for  systems.  Items  scored  as 
concerns  (2)  were  relatively  minor  or  could  be  addressed  through  revisions  to  training  or 
interface  design.  Items  scored  as  deficiencies  (1)  concerned  missing  or  inoperable  functions 
critical  to  the  MDA  mission.  Average  scores  were  computed  for  each  assessment  area.  These 
scores  focused  our  interpretation  of  the  qualitative  data.  We  have  given  special  emphasis  to 
describing  reported  deficiencies. 

It  is  important  to  note  that  many  of  the  reports  are  from  a  single  source  and,  thus,  may  reflect  the 
personal  biases  of  those  sources.  Note  also,  that  the  quantitative  findings  reported  here  were 
developed  by  the  NPS  research  team  from  qualitative  reports,  and  thus  reflect  our  interpretation 
rather  than  the  interpretation  of  diverse  operational  experts  or  technology  experts. 

In  the  next  sections,  we  summarize  our  findings  from  each  of  the  assessment  areas.  Following 
this,  we  present  recommendations  based  on  these  findings,  and  then  describe  the  findings  in 
detail. 

Summary  of  MDA  Capabilities  Results 

This  section  presents  a  qualitative  and  quantitative  summary  of  MDA  capabilities  within  the 
assessment  framework  developed  for  this  effort.  It  summarizes  findings  across  the  assessment 
venues.  The  data  presented  here  comprise  a  compilation  of  194  observations,  warfighter 
comments,  and  survey  results.  These  were  each  assigned  to  one  or  more  of  the  relevant 
assessment  sub-areas.  Many  items  were  placed  in  multiple  categories,  resulting  in  a  total  of  304 
assessments.  Items  were  scored  on  the  three-point  scale  described  above.  Average  scores  drove 
interpretation  of  the  qualitative  data. 

Results  by  Assessment  Area 

The  data  generated  across  the  MDA  assessment  venues  focused  on  the  use  of  MDA  Spiral- 1 
technologies  for  developing  and  tracking  vessels  of  interest  (VOIs)  and  conducting  MIO 
operations.  Accordingly,  as  depicted  in  Table  3,  most  of  the  data  were  categorized  in  the  system 
performance  assessment  area  (219  items)  and  the  operations  performance  assessment  area  (70 
items).  In  both  areas,  the  average  assessment  score  was  relatively  high:  2.4  out  of  3.  No  items 
concerned  warfighter  performance  independent  of  the  MDA  technologies,  and  no  assessment  is 
made  in  this  area.  Only  eight  items  concerned  the  organization/guidance  assessment  area,  and 
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seven  items  concerned  the  system  supportability  and  readiness  area.  Assessment  scores  were 
lowest  in  these  low-frequency  categories,  largely  due  to  overall  concerns  about  the  utility  and 
supportability  of  new  technologies  from  organizations  being  tasked  with  the  new  MDA  mission. 
The  frequency  of  reference  to  Spiral- 1  and  other  MDA  technologies  is  presented  in  Table  3. 

Table  3.  Distribution  of  data  by  assessment  area. 


Area 

Items 

Avg  Score 

Operations  Performance 

70 

2.4 

Organization/Guidance 

8 

2.1 

System  Performance 

219 

2.4 

System  Supportability  and  Readiness 

7 

2.0 

Warfighter  Performance 

0 

- 

Total  /  Average 

304 

2.4 

The  data,  which  include  warfighter  comments,  observations,  interview  notes,  and  survey  results, 
specified  123  strengths  of  the  tested  MDA  systems,  166  concerns,  and  15  deficiencies,  as 
displayed  in  Table  4.  Ten  deficiencies  concerned  System  Perfonnance,  and  five  concerned 
Operations  Performance.  Because  deficiencies  are  the  most  critical  data,  we  address  these  in 
depth  in  the  relevant  assessment  areas,  below. 


Table  4.  Distribution  of  data  by  score  by  assessment  area. 


Count  of  Score 

Score 

Assessment  Area 

1:  Deficiency 

2:  Concern 

3:  Strength 

Total 

Operations  Performance 

5 

33 

32 

70 

Organization/Guidance 

7 

1 

8 

System  Perfonnance 

10 

119 

90 

219 

System  Supportability  and 
Readiness 

- 

7 

- 

7 

Total 

15 

166 

123 

304 

We  turn  now  to  a  qualitative  assessment  of  each  assessment  area  for  which  there  was  data. 

System  Performance 

Of  219  items  that  concerned  System  Performance,  10  specified  potential  deficiencies  regarding 
MDA  system  performance.  Three  of  these  concerned  the  inability  to  specify  baselines  against 
which  to  compare  observed  vessel  behavior.  Two  items  concerned  perceived  inadequacy  of  data 
quality  and  availability  and  two  items  concerned  lack  of  connectivity  of  MIO  technology.  The 
remaining  three  items  concerned  specific  features,  data  sharing,  or  security.  Because  deficiencies 
are  particularly  important  in  assessment,  we  present  the  detailed  comments  regarding  potential 
deficiencies  regarding  system  perfonnance  in  Appendix  D. 

Some  119  items  expressed  reparable  concerns  with  MDA  systems.  By  far  the  largest  group  of 
these  (33  items)  addressed  the  completeness,  correctness,  conflicts,  and  timeliness  of  data  or  data 
processing.  Usability  of  maps,  search,  alerts,  and  other  features  was  cited  in  24  items.  Some  20 
items  addressed  problems  with  training  availability,  fit  to  local  needs,  or  speed.  Operational 
utility  was  a  concern  in  7  cases,  and  the  redundancy  of  technologies  was  an  issue  in  5  items. 
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Of  the  90  strengths  cited  regarding  System  Perfonnance  (Table  4)  across  the  various  types  of 
data  collected  (i.e.,  operator  comments,  observations,  and  survey  results),  27  concerned 
operational  utility  for  VOI  detection  and  tracking  and  for  MIO  operations.  Usability  of  search, 
maps,  and  other  features  accounted  for  18  items.  The  speed  of  training  (6  items)  was  also  an 
indirect  validation  of  the  usability  of  the  technologies.  Nine  items  concerned  the  value  of  data 
fusion  capabilities  of  these  technologies.  The  speed  of  the  technologies  was  cited  in  6  items. 

Operations  Performance 

Of  70  items  that  concerned  Operations  Perfonnance  five  were  potential  deficiencies  (see 
Appendix  E  for  a  list  of  potential  deficiencies  regarding  Operations  Perfonnance).  All  but  one 
item  concerned  E-MIO  connectivity  (reception  or  transmission)  problems.  The  remaining  item 
concerned  gaps  in  track  coverage  by  CMA  due  to  failures  of  CMA  servers.  (Note  that  one  of  the 
five  items  in  the  table  in  Appendix  E  was  coded  in  two  assessment  categories). 

Of  33  items  coded  as  concerns  (i.e.,  a  score  of  2),  the  most  frequent  themes  were  the  complete¬ 
ness,  correctness,  conflicts,  timeliness  of  data  and  data  processing  (12  items),  usability  (10 
items),  and  training  (5  items).  Of  32  items  coded  as  strengths,  13  concerned  operational  utility 
for  VOI;  10  concerned  usability;  4  focused  on  the  utility  of  alerts;  and  the  rest  addressed 
miscellaneous  issues. 

Warfighter  Performance 

No  items  concerned  warfighter  performance  independent  of  the  MDA  technologies,  and  thus  no 
assessment  is  made  in  this  area. 

Organization/Guidance 

Very  few  items  (8  out  of  a  total  of  304)  concerned  the  assessment  area  of  Organization  and 
Guidance.  Of  these,  none  rose  to  the  level  of  a  potential  deficiency.  Seven  items  were  coded  as 
concerns.  Two  each  concerned  the  constraints  imposed  by  policies  and  agreements;  coordination 
of  MDA  execution  and  ONI  intelligence  processes;  and  training.  One  addressed  the  challenge  of 
customizing  solutions  to  fit  the  local  missions  of  organizations.  One  item,  scored  as  a  strength 
(i.e.,  a  score  of  3),  concerned  the  alignment  of  MDA  tasks  with  MHQ  w/  MOC  processes 
documented  in  the  Process  Alignment  Workshop  (Freeman,  Heacox,  &  MacKinnon,  2008). 

System  Supportability  and  Readiness 

Seven  items  addressed  System  Supportability  and  Readiness.  All  of  these  were  areas  of  concern, 
either  regarding  the  competency  or  size  of  staff  (4  items),  or  potential  deficiencies  in  infrastruc¬ 
ture  (2  items)  or  variance  between  facilities  ( 1  item)  that  might  hinder  fielding  and  use  of  MDA 
solutions. 

Results  by  Assessment  Sub-Area 

In  this  section,  we  summarize  findings  in  each  assessment  sub-area.  A  summary  of  the  results  of 
the  quantitative  assessment  is  presented  in  Table  5,  followed  by  a  qualitative  assessment  that 
summarizes  the  strengths,  concerns,  and  deficiencies  observed  in  the  various  MDA  assessment 
venues.  Note,  in  the  quantitative  assessment,  that  areas  of  greatest  concern  (lowest  average 
scores)  were  Operations  Perfonnance  for  MIO,  several  areas  of  Organization/Guidance 
(Guidance,  MDA  Compatibility,  and  System  Management  and  Security),  and  System  Support- 
ability  and  Readiness.  The  most  highly  rated  sub-areas  were  Operations  Performance:  Know- 
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ledge  Processes,  Organization/Guidance:  MHQ/MOC  Compatibility,  and  System  Performance: 
Operations  Support.  Finer-grained  categorization,  a  synopsis,  and  a  score  for  each  item  can  be 
found  in  the  complete  report  on  this  effort  (MacKinnon,  Hutchins,  Schacher,  &  Freeman,  2008). 


Table  5.  Average  assessment  scores  by  sub-area. 


Assessment  Sub- Area 

Average  score 

Operations  Performance:  Knowledge  Processes 

2.5 

Operations  Perfonnance:  MIO 

1.8 

Organization/Guidance:  Agreements 

2.0 

Organization/Guidance:  Guidance 

2.0 

Organization/Guidance:  MDA  Compatibility 

2.0 

Organization/Guidance:  MHQ/MOC  Compatibility 

2.5 

System  Performance:  Automation 

2.4 

System  Performance:  Operations  Support 

2.5 

System  Performance:  System  Management  and  Security 

2.0 

System  Performance:  Technical  Performance 

2.3 

System  Performance:  Warfighter  Acceptance 

2.4 

System  Supportability  and  Readiness 

2.0 

Average 

2.4 

System  Performance:  Technical  Performance 

Table  6  shows  the  majority  of  the  81  items  concerning  System  Performance:  Technical 
Performance  involved  Information  Retrieval  (54  items),  all  but  30  of  which  addressed  concerns 
(i.e.,  score  of  2)  about  the  completeness,  correctness,  conflicts,  timeliness  of  data  and  data 
processing.  Usability  strengths  and  concerns  were  the  topic  of  8  items.  Also  noted,  with  5  items 
each,  were  data  fusion  (generally,  a  strength)  and  speed  of  data  processing  (which  received 
mixed  assessments).  The  remaining  items  addressed  various  issues. 

Table  6.  Number  of  items  concerning  System  Performance:  Technical  Performance. 


Sub-Sub- Area 

Total 

System  Performance:  Technical  Performance:  Infonnation  Processing 

1 

System  Performance:  Technical  Performance:  Infonnation  Retrieval 

54 

System  Performance:  Technical  Performance:  Infonnation  Sharing 

21 

System  Performance:  Technical  Perfonnance:  Interoperability 

5 

Total 

81 

Also  shown  in  Table  6,  of  the  remaining  items,  21  concerned  Information  Sharing.  More 
specifically,  7  addressed  the  completeness,  correctness,  conflicts,  timeliness  of  data  and  data 
processing.  Three  each  concerned  data  sharing  and  transmission.  Five  items  addressed 
Interoperability  of  systems.  Of  these  items,  three  addressed  requirements  for  specific  software 
(e.g.,  Java,  Direct-X),  and  two  concerned  the  availability  of  data  within  or  between  nodes.  One 
item  addressed  Infonnation  Processing,  specifically  the  need  for  automated  statistical  analysis  of 
data  gaps  to  identify  potential  new  sources  of  information  and  drive  new  collections.  This 
capability  -  data-driven  collection  planning  -  is  being  developed  in  research  efforts  for  ground 
warfare,  and  could  be  extended  to  MDA. 
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System  Performance:  Operations  Support 

Of  21  items  that  concerned  System  Perfonnance:  Operations  Support,  20  concerned  the  utility  of 
specific  technologies  to  operations.  As  indicated  by  comments  included  in  Appendices  D  and  E, 
E-MIO  technology  exhibited  a  potential  deficiency  with  respect  to  connectivity.  Most  items  in 
this  area  were  strengths  or  concerns  about  the  general  value  of  these  tools.  One  item  concerned 
the  lack  of  tactics,  techniques,  and  procedures  (TTPs)  and  standard  operating  procedures  (SOP) 
for  applying  MDA  technologies. 

System  Performance:  Warfighter  Acceptance 

Some  85  items  addressed  Warfighter  Acceptance  of  specific  technologies.  Of  these,  41 
concerned  Human-System  Interaction,  the  majority  focusing  on  usability  of  agents,  alerts,  briefs, 
help,  maps,  menus,  and  search.  Five  of  these  concerned  CMA’s  data  fusion  capabilities,  which 
were  viewed  as  a  strength  in  most  cases.  Training  was  addressed  in  22  items,  which  was  seen  as 
fast  but  often  incomplete.  System  Utility  was  addressed  by  18  items,  with  an  emphasis  on 
acceleration  of  operational  tasks,  and  benefits  to  situational  awareness.  However,  MDA  DS  COI, 
LiNX,  and  Google  Earth  were  each  viewed  once  with  concern  (i.e.,  a  score  of  2)  for  being 
redundant  with  other  solutions.  Strategies  for  using  the  systems  were  seen  to  be  lacking  in  four 
cases  in  an  area  we  call  System  Usage. 

System  Performance:  Automation 

Automation  was  addressed  in  27  items.  Of  these,  26  concerned  Alerts,  specifically  their  strong 
operational  utility  but  mixed  usability.  One  item,  concerning  Infonnation  Processing,  addressed 
MASTER’S  strong  automation  to  capture  data  on  VOIs,  relative  to  current  solutions. 

System  Performance:  System  Management  and  Security 

Of  five  System  Management  and  Security  issues,  4  were  concerns  about  keeping  systems  up  and 
running,  and  one  -  a  potential  deficiency  -  concerned  loss  of  the  original  security  classification 
of  information  to  be  disseminated. 

Operations  Performance:  Knowledge  Processes 

VOI  Development  was  the  focus  of  52  of  the  62  items  concerning  Operations  Performance: 
Knowledge  Processes.  In  this  area,  usability  (17  items),  operational  utility  (14  items),  and  the 
completeness,  correctness,  conflicts,  timeliness  of  data  and  data  processing  were  the  most 
frequent  topics.  VOI  Tracking  accounted  for  the  remaining  10  items,  and  similar  concerns  arose 
in  this  area;  completeness,  correctness,  conflicts,  timeliness  of  data  and  data  processing  (4 
items),  and  mixed  usability  (2  items)  were  the  most  frequently  cited  issues. 

Operations  Performance:  MIO 

In  the  area  of  MIO  Execution,  inadequate  training  was  a  concern  in  3  of  8  cases;  inadequate 
connectivity  was  a  potential  deficiency  in  2. 

Organization/Guidance:  MDA  Compatibility 

The  alignment  of  ONI  with  MDA  activities  was  a  concern  in  2  items. 

Organization/Guidance:  MHQ/MOC  Compatibility 

Two  items  concerned  compatibility  of  MDA  and  MHQ  with  MOC.  Process  Alignment  between 
MDA  and  MHQ  with  MOC  was  seen  as  a  strength  in  one  item,  due  to  the  Process  Alignment 
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Workshop.  However,  local  Organization  Alignment  with  MDA,  that  is,  the  flexibility  of  MDA 
processes  to  local  needs,  was  a  concern  in  one  case. 

Organization/ Guidance :  Agreements 

Two  items  concerned  the  assessment  area  Organization/Guidance:  Agreements,  and  both  were 
concerns  (i.e.,  rating  of  2)  about  restrictions  imposed  by  data  sharing  agreements  about  classified 
information  or  data  concerning  U.S.  citizens. 

Organization/Guidance:  Guidance 

Two  items  specified  concerns  (i.e.,  rating  of  2)  about  the  need  to  develop  better  MDA  CONOPS 
and  TTP/SOP. 

System  Supportability  and  Readiness 

Seven  items  concerned  System  Supportability  and  Readiness.  Of  these,  4  were  concerns  (i.e., 
rating  of  2)  about  the  number  or  competency  of  staff,  and  3  identified  concerns  about  the 
adequacy  of  infrastructure  to  support  MDA  technologies  generally. 

Results  by  Technology 

In  this  section,  we  summarize  the  scores  for  each  MDA  technology  cited  in  the  findings  - 
including  Spiral- 1  technologies  and  others  -  at  the  lowest  assessment  level.  For  further  detail 
concerning  each  technology,  we  refer  the  reader  to  the  sections  below,  in  which  we  present 
findings  in  each  of  the  assessment  areas. 

The  majority  of  the  data  gathered  across  venues  concerned  a  specific  technology  (a  total  of  257 
items),  usually  CMA  (136  items).  A  minority  of  the  data  concerned  no  specific  MDA 
technology,  or  an  unspecified  suite  of  technologies  (47  items)  (see  Table  7).  Note  that  the  list  of 
technologies  includes  some  systems  that  are  not  Spiral- 1  products,  but  that  were  evaluated  in  the 
various  MDA  assessment  venues. 

Those  technologies  that  scored  lowest,  on  the  average  (see  Table  7),  were  Global  Trader,  for 
perceived  incompleteness  of  data  and  lack  of  data  replication;  MAGNET,  for  lack  of  data  due  to 
data  sharing  agreements  concerning  U.S.  citizens;  MDA  DS  COI  for  lack  of  usability  or  training; 
and  E-MIO  wireless  for  connectivity  problems.  Those  that  scored  highest  were  Tripwire,  for  its 
alerting  and  provision  of  contextual  information,  and  PANDA,  for  the  operational  utility  of  alerts 
and  quality  of  explanations  of  deviations  that  it  presented. 


Table  7.  Distribution  of  data  by  technology. 


Technology 

Frequency 

Average  Score 

All  Spiral- 1 

47 

2.1 

CMA 

136 

2.4 

E-MIO  Wireless 

23 

2.1 

FASTC2AP 

13 

2.4 

Global  Trader 

5 

2.2 

Google  Earth 

10 

2.5 

LiNX 

5 

2.4 

MAGNET 

2 

2.0 

12 


MASTER 

33 

2.3 

MDA  DS  COI 

5 

2.0 

MIDAS 

5 

2.6 

PANDA 

13 

2.9 

Tripwire 

7 

2.9 

Total/  Avg 

304 

2.4 

A  more  detailed  assessment  is  presented  in  Appendix  F,  which  contains  the  average  assessment 
scores  for  MDA  technologies  at  the  finest  level  of  assessment  detail. 

To  make  a  more  focused  assessment  of  the  operational  utility  of  systems,  we  computed  the  aver¬ 
age  sum  of  scores  for  strengths  and  concerns1  about  systems.  Several  MDA  technologies  appear 
to  have  particularly  high  or  low  operational  utility,  based  on  this  analysis  (see  Table  8). 
FASTC2AP,  Global  Trader,  PANDA,  Tripwire,  and  CMA  received  high  marks  for  operational 
utility.  MDA  DS  COI,  E-MIO  Wireless,  and  MAGNET  received  low  marks  for  operational 
utility. 

Table  8.  Average  Sum  of  Concern  and  Strength  Scores  by  Technology. 


MDA  Technology 

Avg  Sum  of  Scores 

FASTC2AP 

3.0 

Global  Trader 

3.0 

PANDA 

3.0 

Tripwire 

3.0 

CMA 

2.8 

MASTER 

2.7 

Google  Earth 

2.5 

LiNX 

2.3 

MDA  DS  COI 

2.0 

E-MIO  Wireless 

1.5 

MAGNET 

n.a. 

An  additional  analysis  was  conducted  to  help  the  reader  assess  the  impact  of  specific  systems 
regarding  fulfillment  of  MDA  capability  requirements  (see  Table  9).  Spiral- 1  technologies  must 
help  the  Navy  to  fulfill  several  MDA  capabilities:  monitor,  collect,  fuse,  analyze,  and  dissemi¬ 
nate.  One  technology,  E-MIO  Wireless,  has  a  low  assessment  score  (Table  8,  driven  by  connec¬ 
tivity  deficiencies)  and  it  addresses  only  one  MDA  capability  (i.e.,  collect).  This  should  raise 
concern  about  the  value  of  this  technology,  and  focus  investment  (or  disinvestment)  decisions  on 
it. 

Low  assessment  scores  (of  2.0,  or  less),  in  Table  9,  are  also  a  concern  with  respect  to  MAGNET 
and  MDA  DS  COI.  However,  these  technologies  address  multiple  MDA  capabilities.  Strengths 
in  one  area  may  compensate  for  concerns  in  others. 


1  The  value  here  is  computed  as  the  sum  of  all  scores  for  concerns  (each  valued  at  2  points)  and  strengths  (3  points) 
divided  by  the  total  frequency  of  these  scores  for  items  in  these  categories:  System  Performance  categories  for 
Operations  Support:  System  Utility,  and  Warfighter  Acceptance:  System  Utility.  Note  that  scores  for  potential 
deficiencies  were  excluded  from  this  analysis,  as  were  scores  in  all  other  assessment  categories. 
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Note  that  all  MDA  capabilities  are  addressed  by  more  than  one  technology.  Thus,  the  issues 
identified  with  respect  to  any  one  technology  do  not  necessarily  indicate  a  capability  gap.  More 
specifically,  three  SP-1  technologies  (CMA,  MAGNET,  and  Tripwire)  address  all  of  these 
capabilities.  Three  technologies  (Global  Trader,  DS/COI,  and  FASTC2AP)  address  most 
capabilities.  Three  technologies  address  only  one  capability  (E-MIO:  Collect;  Google  Earth: 
Disseminate;  and  LiNX:  Collect).  A  finer-grained  understanding  of  the  implications  of  findings 
on  MDA  capabilities  can  be  had  by  considering  the  mapping  of  technologies  to  the  performance 
thresholds  that  each  technology  must  achieve,  as  presented  in  Appendix  H. 

Table  9.  Mapping  of  SP-1  Technologies  to  MDA  Capabilities. 


Capability 

CMA 

E-MIO 

Wireless 

Global 

Trader 

Google 

Earth 

LiNX 

MAGNET 

Data 

Sharing 

COI 

Tripwire 

FASTC2AP 

Assessment 

Score 

2.4 

2.1 

2.2 

2.5 

2.4 

2.0 

2.0 

2.9 

2.4 

Monitor 

X 

X 

X 

X 

X 

X 

Collect 

X 

X 

X 

X 

X 

X 

X 

X 

Fuse 

X 

X 

X 

X 

Analyze 

X 

X 

X 

X 

X 

Disseminate 

X 

X 

X 

X 

X 

X 

X 

MDA  Capabilities  Recommendations 

In  this  section,  we  make  recommendations  based  on  the  findings,  discussed  in  previous  sections. 
Recommendations  concern  fielding  and  support,  technology  usability,  technology  functionality, 
data  quality,  training,  organizational  interoperability,  and  future  MDA  assessment  activity.  These 
recommendations  are  organized  into  three  categories  of  the  assessment  framework,  and  an 
additional  category  concerning  the  programmatics  of  MDA  assessment. 

System  Supportability  and  Readiness 

Strong  technologies’.  PEO  C4I  and  Office  of  the  Chief  of  Naval  Operations  (OPNAV)  should 
press  forward  with  development  and  fielding  of  several  technologies  that  received  high  marks  for 
operational  utility:  FASTC2AP,  Global  Trader,  PANDA,  Tripwire,  and  CMA.2 

Weaker  technologies :  OPNAV  should  evaluate  the  expressed  concerns,  deficits,  and  return  on 
investments  for  several  technologies  that  received  low  marks  for  operational  utility:  MDA  DS 
COI,  E-MIO  Wireless,  and  MAGNET. 

Fielding  sites:  OPNAV  should  consider  placing  technologies  in  reach-back  facilities  (e.g.,  at 
ONI  rather  than  at  NAVCENT)  with  robust  technical  support  and  operator  competence  for  those 
technologies  that  have  high  utility  but  low  accuracy,  reliability,  or  usability. 


2  This  list  was  determined  by  computing  the  average  weighted  sum  of  scores  for  items  in  two  categories  concerning 
operational  utility:  System  Performance  categories  for  Operations  Support:  System  Utility,  and  Warfighter 
Acceptance:  System  Utility.  Scores  for  potential  deficiencies  were  excluded  from  this  analysis. 
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Fielding/support  process :  OPNAV  and  PEO  C4I  should  ensure  and  advertise  competent  fielding 
and  support  for  MDA  technologies.  This  may  involve  educating  these  organizations  about  the 
processes  for  surveying  infrastructure  (e.g.,  the  adequacy  of  power  supplies  and  server  space), 
configuring  or  customizing  technologies  to  local  needs,  specifying  technical  support 
requirements,  fulfilling  them,  and  identifying  and  addressing  shortfalls. 

System  Performance 

Usability:  OPNAV  and  PEO  C4I  should  systematically  analyze  the  usability  of  MDA 
technologies  -  specifically  programmable  agents,  maps,  search  features,  graphs,  alerts,  and 
briefing  products  -  to  ensure  that  (1)  the  cost  in  errors  and  response  time  is  estimated,  (2)  design 
modifications  are  prioritized  accordingly  and  are  funded,  and  (3)  training  enables  operators  to 
work  around  persistent  usability  problems.  In  MDA  assessment  events,  CMA,  MASTER, 
FASTC2AP,  MDA  DS  COI,  and  E-MIO  all  had  usability  issues  that  raised  concern. 

Redundant  functionality:  OPNAV  and  PEO  C4I  should  evaluate  reports  of  redundancy  of  MDA 
technologies  with  each  other  and  with  existing  systems.  Fielding  decisions,  tactics,  techniques, 
and  procedures  (TTPs),  and  training  should  resolve  these  redundancies. 

Baseline  specification  functions:  OPNAV  and  PEO  C4I  should  ensure  that  technologies  with 
alert/alann  capabilities  enable  the  user  to  specify  baseline  behaviors  for  traffic  in  different 
regions.  Without  this  function,  alerts  lose  much  of  their  value. 

Data  sources:  OPNAV  should  identify,  develop,  and  link  to  critical  data  sources.  For  example, 
data  sharing  agreements  are  required  to  provide  data  concerning  U.S.  persons. 

Data  source  education:  OPNAV  and  PEO  C4I  should  ensure  operators  of  new  systems 
understand  which  data  sources  feed  those  systems,  the  reliability  of  those  sources,  which 
expected  data  sources  do  not  feed  those  systems,  and  the  implications  of  both  for  analyzing 
results.  These  explanations  should  identify  the  reason  for  data  gaps,  e.g.,  lack  of  institutional  or 
international  agreements  to  access  data,  lack  of  a  connection  to  the  data  source,  lack  of  data 
replication,  or  data  loss  during  system  failures.  In  MDA  assessment  events,  unreliable  sources  or 
systems  produced  gaps  in  CMA  track  coverage;  lack  of  data  replication  produced  incomplete 
data  in  Global  Trader;  connectivity  issues  hindered  use  of  E-MIO  (e.g.,  below  decks);  and 
MAGNET  data  were  sparse  per  policies  about  handling  data  concerning  U.S.  persons. 

Common  Data:  OPNAV  should  analyze  and  resolve  significant  differences  in  data  sets  between 
users  who  must  coordinate  their  activities. 

Training:  OPNAV  should  implement  a  dedicated  course  of  instruction  concerning  MDA  TTPs, 
standard  operating  procedures  (SOPs),  and  the  role  of  MDA  technologies  in  them,  and  include  it 
in  the  Navy  Training  System  Plan  (NTSP).  Ideally,  this  training  will  be  customized  to  local 
missions  and  conditions. 

Organization/Guidance 

Process  interoperability:  OPNAV  should  evaluate  and  revise  the  emerging  process  architectures 
for  MHQ  with  MOC,  ONI,  and  other  organizations  to  ensure  they  support  MDA  tasks,  and  they 
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can  be  customized  to  the  conditions  of  MDA  work  in  the  varied  Navy  areas  of  operations.  This 
analysis  may  entail  developing  MDA  use  cases  or  scenarios  against  which  to  test  process 
architectures,  and  these  use  cases  can  serve  double  duty  as  exercise  and  training  scenarios. 

Information  flow  impedance:  OPNAV  and  PEO  C4I  should  assess  the  impact  of  new 
technologies,  procedures,  and  organizational  structures  on  the  rate  of  information  flow  between 
organizations,  and  between  elements  of  organizations.  New  technologies  have  the  potential  to 
raise  the  infonnation  output  of  some  organizations  (e.g.,  ONI)  well  above  the  input  and 
processing  capacities  of  the  organizations  they  support  (e.g.,  NAVCENT). 

Assessment 

MDA  operational  baseline :  To  estimate  the  return  on  investment  in  MDA  systems  requires  that 
we  define  the  baseline  of  MDA  capabilities  (e.g.,  the  number  of  VOIs  developed  and  tracked  per 
unit  time).  This  baseline  might  be  estimated  by  operational  experts.  However,  more  reliable  data 
will  be  obtained  from  direct  observation  of  the  effectiveness  of  current,  fielded  technologies  for 
MDA  missions.  We  recommend  that  observational  data  be  gathered  concerning  the  current  MDA 
baseline. 

MDA  system  alternatives:  The  return  on  investment  for  MDA  assessment  may  be  increased  if 
experimentation  and  observation  address  some  promising  systems  outside  the  Spiral- 1  suite. 

Such  systems  include  CMA,  NEPTUNE3,  GALE -Lite,  Palaemon,  PANDA,  and  Sea  Watch. 
Evaluation  of  such  systems  is  essentially  a  high-risk/high- return  investment  in  the  portfolio  of  MDA 
assessment  activities. 

MDA  assessment  scenarios:  OPNAV  and  PEO  C4I  should  develop  exercises  that  train  and  test 
MDA  capabilities  (technologies,  TTPs,  organizations,  etc.),  with  a  particular  focus  on  handling 
realistic  numbers  of  white  vessels.  Design  these  exercises  to  answer  at  least  these  questions:  Can 
we  develop  a  VOI  and  identify  it  among  many  vessels  almost  identical  to  it?  Can  we  communi¬ 
cate  securely  and  confidentially  with  a  single  ship  among  many  ships  in  an  area?  Can  we 
coordinate  our  actions  in  a  crowded  field  of  internationally  flagged  white  vessels?  Can  we 
manage  and  address  costs  of  delay  to  commercial  shipping  (estimated  by  NORTHCOM  at 
$  10,000/hour)  and  erroneous  actions  (e.g.,  damage  or  destruction  of  internationally  flagged 
vessels)? 

MDA  process  modeling:  OPNAV  should  ensure  that  future  assessments  use  measures  that 
support  computational,  “what  if’  modeling  of  the  impact  of  new  technologies,  processes, 
manning,  and  organizational  structures.  Such  measures  should  represent  the  speed-accuracy 
tradeoff  curve  for  analysis,  decision  making,  and  action  (throughput)  given  varied  missions,  staff 
size  and  competency,  and  related  factors.  By  reusing  these  measurements  in  models,  the  Navy 
will  multiply  the  answers  it  can  extract  from  scant  assessment  data,  and  thus  increase  its  return 
on  every  dollar  spent  on  assessment. 

System  Performance  Results 

The  assessment  area  System  Performance  concerns  the  performance  of  MDA  technologies, 
including  Spiral- 1  technologies.  The  components  of  this  assessment  area  concern  the  technical 


3  NEPTUNE,  GALE-Lite,  Palaemon,  and  PANDA  are  systems  being  developed  to  support  MDA  missions  that  are 
not  part  of  the  Spiral- 1  set  of  technologies. 
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performance  of  systems,  the  support  they  provide  to  operations,  warfighter  acceptance  of  the 
systems,  automation  of  important  functions,  and  system  management  and  security. 

Organization/Guidance  Results 

The  assessment  area  Organization/Guidance  concerns  the  alignment  of  organizations  to  MDA 
missions,  the  alignment  of  MDA  processes  or  tasks  with  those  specified  in  the  MHQ  with  MOC 
process  architecture  the  alignment  of  international  and  inter-organization  agreements  with  MDA 
mission  requirements,  and  the  availability  of  CONOPS,  TTPs,  SOP,  and  standing  orders  for 
MDA  missions.  The  key  data  concerning  this  assessment  area  are  presented  in  Appendix  H. 

Observer  Log  Results 

The  following  results  are  for  CMA,  except  where  noted. 

Information  Acquisition  Times  -  When  data  on  information  acquisition  times  was  collected, 
although  the  systems  were  not  being  used  in  an  operational  environment,  the  amount  of  time 
required  to  obtain  information  was  viewed  as  representative  of  what  would  be  observed  in 
operational  environments.  Very  little  time  was  required,  e.g.,  1  min  to  detennine  when  a 
specified  ship  was  built  after  the  White  Cell  request  for  information;  7  min  to  detennine  a 
person-of- interest  was  not  on  the  reported  ship;  2  min  to  identify  the  type  of  cargo  on  a  named 
ship;  10  min  to  locate  a  ship  and  identify  its  origin  and  anticipated  arrival  locations;  and,  6  min 
to  locate  a  ship,  details  on  type  and  crew. 

It  is  possible  that  these  times  are  biased  to  shorter  time  because  historical  data  was  used  for 
which  the  needed  data  was  known  to  be  present.  Even  so,  these  information  acquisition  times 
represent  an  improvement  over  existing  times. 

System  Performance  -  Most  system  problems  encountered  were  due  to  the  lab  environment  and 
do  not  qualify  as  Spiral- 1  system  results.  The  following  results  are  inherent  to  the  systems,  not 
the  lab  environment.  These  results  are  derived  from  limited  tests  of  the  systems  and  have  not 
been  validated.  (1)  Partial  match  did  not  work  as  anticipated  when  searching  for  a  ship  name.  (2) 
Operators  could  not  zoom  in  and  draw  a  small  geographical  box  around  a  VOI. 

Summary  of  Findings 

Most  of  the  data  reported  here  describe  strengths  or  concerns  of  MDA  systems.  Few  users 
identified  deficiencies  in  these  systems.  The  scores  indicate  all  of  the  assessed  technologies 
increased  the  warfighter’s  effectiveness  to  some  degree,  yet  there  was  little  information  about  the 
level  of  enhancement  of  MDA  capabilities  provided  by  the  suite  of  Spiral- 1  systems  due  to  the 
lack  of  data  on  baseline  levels  of  perfonnance. 

Specifically,  in  the  quantitative  assessment,  areas  of  greatest  concern  (lowest  average  scores)  were: 

>  Operations  Perfonnance  for  EM  10* 

>  Several  sub-areas  of  Organization/Guidance  concerning  MDA  Compatibility 
(specifically,  alignment  of  MDA  activity  with  ONI) 

>  Guidance  (especially  the  need  for  MDA  CONOPS,  TTPs,  and  SOPs) 

>  Agreements  (concerning  data  sharing) 

>  System  Management 

>  Security 

>  System  Supportability 
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>  Readiness 

*  Assessment  for  EMIO  was  obtained  through  an  actual  operational  experiment,  not  a  system 
test.  This  differs  from  the  rest  of  the  results  which  were  obtained  from  structured  laboratory 
events. 

The  most  highly  rated  sub-areas  were: 

>  Operations  Perfonnance  concerning  Knowledge  Processes  for  vessel  of  interest 
(VOI  )  development  and  tracking 

>  Organization/Guidance  concerning  Maritime  Headquarters  with  Maritime 
Operations  Center  (MHQ/MOC)  Compatibility  with  MDA 

>  System  Performance  concerning  the  Operations  Support  provided  by  these 
systems. 

Finer  categorization,  a  synopsis,  and  a  score  for  each  item  are  presented  in  the  technical  report  on 
this  effort  (MacKinnon,  Hutchins,  Schacher,  &  Freeman,  2008). 

Future  Assessment  Recommendations 

We  recommend  that  observations  be  conducted  to  establish  a  baseline  on  MDA  capabilities 
using  current  systems,  in  order  to  better  estimate  the  impact  of  Spiral- 1  systems.  We  recommend 
that  some  experimentation  and  observation  address  other  technologies  that  show  promise  to 
improve  MDA  effectiveness.  Such  systems  include  CMA,  NEPTUNE,  GALE -Lite,  Palaemon, 
PANDA,  and  Sea  Watch.  OPNAV  and  PEO  C4I  should  develop  exercises  that  train  and  test 
MDA  capabilities  (technologies,  TTPs,  organizations,  etc.),  with  a  particular  focus  on  handling 
realistically  large  numbers  of  cooperative,  intentionally  uncooperative,  and  inattentive  white 
vessels.  Finally,  we  advise  OPNAV  to  ensure  future  assessments  capture  data  that  support 
computational,  “what  if’  modeling  of  the  impact  of  new  technologies,  processes,  manning,  and 
organizational  structures.  Such  measurements  will  enable  the  Navy  to  reuse  scant  data 
productively  and  in  a  predictive  manner,  thus  multiplying  the  return  on  each  assessment  dollar. 
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Appendix  A:  MDA  Assessment  Areas  and  Structure 


System  Performance 

Technical  Perfonnance 

Information  Retrieval 

Information  Processing 

Information  Sharing 

Operator  Configurable 

Interoperability 

Operations  Performance 

Knowledge  Processes 

Infonnation  Retrieval 

Vol  Development 

Vol  Tracking 

Information  Sharing 

ISR 

Planning 

Execution 

PED 

Operations  Support 

System  Utility 

Standards  and  Guidelines 

Warfighter  Acceptance 

MIO 

System  Utility 

Planning 

Human- System  Interaction 

Execution 

System  Usage 

Assessment 

System  Training 

Automation 

Organization/Guidance 

Alerts 

MDA  Compatibility 

Information  Processing 

Organization  Alignment 

Smart  Pull 

Process  Alignment 

System  Management  and  Security 

MHQ/MOC  Compatibility 

Organization  Alignment 

Warfighter  Performance 

Process  Alignment 

MDA  Mission 

Agreements 

Mission  Understanding 

Information  Sharing 

Mission  Acceptance 

Shared  Operations 

Unit  Performance 

Guidance 

Manning 

CONOPS 

Activities 

TTP/SOP 

Training 

Standing  Orders 

Human  Perfonnance 

Tasks 

System  Supportability  and  Readiness 

Training 

PEO  Provided 
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Appendix  B:  MDA  Assessment  Areas  and  Associated  Metrics 


This  table  presents  the  measures  that  are  to  be  evaluated  for  each  Assessment  Area.  Attributes  are  in 
bold,  followed  by  their  associated  measures. 


MDA  Assessment  Metrics 
MOE 

MOP  or  MOU 


Spiral-1  System  Performance  (each  system) 


Technical  Performance 

Improved:  -5  to  +5  rating  of  improvement  over  existing  systems,  by  system  aspect. 

Needed:  system  fills  a  gap  in  existing  capabilities,  Y/N. 

Applicable:  system  is  applicable  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  available. 

Information 

Accessible:  roll-up  of  information  accessibility. 

Retrieval 

Available:  %  of  time  information  is  available. 

Efficient:  number  of  steps  to  access  information. 

Capable:  roll-up  of  capability  to  retrieve  required  information. 

Sufficient:  %  of  information  needed  for  assessment. 

Timely:  time  required  to  retrieve  information. 

Reliable:  roll-up  of  ability  to  obtain  correct  information  when  needed. 

Assured:  information  source  is  identified,  Y/N. 

Robust:  automatic  failover  during  system  problems,  Y/N;  database  backup,  Y/N. 

Persistent:  %  of  time  down  due  to  system  failure. 

Usable:  roll-up  of  information  usability  for  assessment  and  decision-making. 

Clear:  1-5  rating  of  information  clarity;  1-5  rating  of  GUI  presentation. 

Trusted:  1-5  rating  of  confidence  in  information. 

Information 

Capable:  roll-up  of  ability  to  process  ship  and  Vol  information. 

Processing 

Available:  information  processing  capabilities,  Y/N;  list  capabilities. 

Efficient:  use  of  information  processing  capabilities,  by  capability,  Y/N. 

Sufficient:  information  processing  capabilities  for  operations  needs,  Y/N. 

Automatic:  automatic  information  processing  available?  Y/N 

Information 

Accessible:  roll-up  of  information  sharing  accessibility. 

Sharing 

Compatible:  M2M  interoperability,  by  system,  Y/N. 

Available:  %  of  time  information  sharing  available. 

Efficient:  information  sharing  efficient,  Y/N;  number  of  steps  required  to  share  information. 

Capable:  collaboration  capabilities  provided,  Y/N;  roll-up  of  capability  to  share  information  with  other 
units. 

Reach:  number  of  units  with  which  information  can  be  shared;  number  of  units  per  collaboration 
session. 

Sufficient:  %  of  required  units  with  which  information  can  be  shared;  %  of  required  information 
that  can  be  shared. 

Timely:  time  required  to  exchange  information. 

Operator 

Capable:  1-5  rating  of  the  ability  of  operator  to  configure  the  system  as  desired. 

Configurable 

Flexible:  Operator  can  configure  information  search,  information  presentation,  Y/N. 

Sufficient:  fraction  of  required  profile  types  that  can  be  developed. 

Reach:  Number  of  profiles  that  can  be  saved. 

Efficient:  1  -5  rating  of  system  configuration  efficiency;  number  of  steps  required  to  configure 
system,  by  configuration  type. 

Interoperability 

Compatible:  M2M  system  interoperability,  by  system,  Y/N; 
information  formats  compatibility,  by  system,  Y/N. 

Operations  Support 

System  Utility 

Improved:  -5  to  +5  rating  of  improvement  of  MDA  operation  activities  over  existing  systems,  by 
activity. 

Needed:  system  fills  a  gap  in  existing  support  to  MDA  operations,  Y/N. 

Applicable:  system  is  applicable  to  MDA  activities,  by  activity,  Y/N. 

Wanted:  -5  to  +5  rating  of  operations  center  desire  to  have  system  available. 
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Standards  and 

Guidelines 

Usable:  roll-up  assessment  of  Standards  and  Guidelines  for  system  usage. 

Sufficient:  %  of  system  operations  covered  by  guidelines;  standards  to  cover  information 
formats,  Y/N;  standards  to  cover  M2M  interactions,  Y/N. 

Clear:  guidelines  to  direct  system  operation,  Y/N. 

Prepare  a  list  of  those  situations  for  which  Standards/Guidelines  are  inadequate. 

Warfighter  Acceptance 

System  Utility 

Improved:  -5  to  +5  rating  of  improvement  of  operator's  task  performance  over  existing  systems, 
by  task. 

Needed:  system  fills  a  gap  in  existing  support  to  task  performance,  Y/N. 

Applicable:  system  is  applicable  to  MDA  tasks,  by  task,  Y/N. 

Wanted:  -5  to  +5  rating  of  operator  desire  to  have  system  available. 

Human-System 

Interaction 

Usable:  1-5  scale  roll-up  of  human-system-interaction. 

Clear:  1-5  scale  on  GUI  presentation. 

Manageable:  GUI  can  be  configured  to  operator  desired  presentation,  Y/N. 

Relevant:  information  presented  is  relevant  to  operator  task  performance,  Y/N. 

Timely:  GUI  layout  and  information  presentation  facilitates  rapid  retrieval  of  needed  information, 
Y/N;  time  to  retrieve  needed  information. 

Efficient:  number  of  steps  required  to  retrieve  needed  information;  1-5  scale  on  GUI  facilitation 
of  information  retrieval  efficiency. 

System  Usage 

Prepare  a  table  of  frequency  of  Spiral-1  system  use,  by  situation  and  by  task. 

System  Training 

Usable:  1-5  scale  roll-up  of  training  quality. 

Clear:  1-5  scale  on  training  clarity. 

Sufficient:  1-5  scale  on  sufficiency  of  training  to  prepare  operator  for  tasks. 

Relevant:  1-5  scale  on  whether  training  is  relevant  to  mission  and  workflow. 

Automation 

Alerts 

Capable:  alerts  are  provided,  Y/N;  roll-up  of  quality  of  alerts. 

Automatic: 

Flexible:  alerts  operator  configurable,  Y/N;  number  of  different  types  of  alerts  available. 

Trusted:  1-5  rating  of  alerts  eliminating  need  to  monitor  situation. 

Sufficient:  %  of  needed  alerts  provided  by  system,  %  of  needed  alert  types  provided  by  system. 
Timely:  alerts  provided  in  time  to  take  needed  actions,  Y/N. 

Efficient:  1  -5  rating  of  efficiency  setting  up  alerts;  number  of  steps  required  to  set  up  alerts. 

Information 

Processing 

Capable:  machine  assisted  information  processing  provided,  Y/N;  system  automated  information 
processing  available,  Y/N;  roll-up  assessment  of  information  processing  capability. 

Automatic:  hands-off  information  processing  available,  Y/N. 

Flexible:  system  information  processing  operator  configurable,  Y/N. 

Trusted:  1-5  rating  of  ability  to  accept  hands-off  information  processing. 

Sufficient:  %  of  information  processing  requirements  performed  by  system. 

Smart  Pull 

Capable:  smart  pull  capable,  Y/N. 

Automatic:  smart  pull  automatic  updates,  Y/N. 

Flexible:  smart  pull  operator  configurable,  Y/N;  number  of  different  types  of  smart  pull  available. 
Trusted:  1-5  rating  of  smart  pull  reliability  to  provide  required  information. 

Sufficient:  %  of  required  information  available  by  smart  pull. 

System  Management  and  Security 

Accessible:  roll  up  of  system  management  functions  for  overall  accessibility.  List  any  significant 
causes  for  lack  in  accessibility. 

Reliable:  %  of  time  system  is  down. 

Secure:  %  of  attacks  that  disrupt  system  performance. 

Manageable:  time  required  to  repair/reconfigure  system  after  failure. 

Sufficient:  information  in  status  reports  to  manage  system,  Y/N. 

Accurate:  %  of  system  status  reports  that  are  correct. 

Operations  Performance  , 

Knowledge  Processes 
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Information 

Retrieval 

Accurate:  %  of  retrieved  information  that  conforms  to  ground  truth. 

Flexible:  number  of  sources  that  can  be  accessed  to  provide  information,  by  information  type. 
Sufficient:  information  available  to  assess  vessels  in  AoR;  information  available  to  conduct  MDA 
operations; 

Timely:  time  to  retrieve  information,  by  information  type;  information  in  time  to  take  needed 
actions. 

Efficient:  1-5  rating  of  information  retrieval  efficiency,  by  information  type;  Number  of  steps 
required  to  retrieve  information,  by  information  type. 

Vol 

Development 

Accurate:  %  of  correct  assessments  of  vessel  classification,  threat. 

Timely:  time  to  complete  Vol  assessment  and  classification. 

Vol  Tracking 

Capable:  1-5  rating  of  ability  to  track  vessels;  roll-up  summary  of  vessel  tracking  capabilities. 

Accurate:  mean  vessel  location  error,  by  vessel  type. 

Flexible:  number  of  different  types  of  vessels  that  can  be  tracked  by  (radiating,  AIS,  etc) 

Reach:  geographical  area  over  which  tracking  can  be  accomplished. 

Sufficient:  %  of  AOR  over  which  tracking  can  be  accomplished. 

Timely:  time  to  locate  vessel;  frequency  of  vessel  reports. 

Information 

Sharing 

Capable:  collaboration  capabilities  provided,  Y/N;  roll-up  of  capability  to  share  information  with  other 
units. 

Reach:  number  of  units  with  which  information  can  be  shared;  number  of  units  per  collaboration 
session. 

Sufficient:  %  of  required  units  with  which  information  can  be  shared;  %  of  required  information 
that  can  be  shared. 

Timely:  time  required  to  exchange  information,  ops  centers  and  reachback. 

Compatible:  system  interoperability,  Y/N;  information  formats,  by  unit,  Y/N. 

Accessible:  roll-up  of  information  sharing  accessibility. 

Compatible:  M2M  interoperability,  by  system,  Y/N. 

Available:  %  of  time  information  sharing  available. 

Efficient:  information  sharing  efficient,  Y/N;  number  of  steps  required  to  share  information. 

ISR 

Planning 

Sufficient:  %  of  RFIs  addressed;  %  of  available  assets  assigned. 

Timely:  time  to  plan;  planning  completed  in  time  for  execution. 

Execution 

Accurate:  %  of  assets  conforming  to  planed  actions. 

Sufficient:  %  of  assets  completing  assignment. 

PED 

Accurate:  %  of  assessments  conforming  to  ground  truth. 

Timely:  time  to  complete  processing,  exploitation,  distribution;  distribution  in  time  to  meet 
planning  cycle. 

MIO 

Planning 

Sufficient:  %  of  RFIs  addressed;  %  of  available  assets  assigned. 

Timely:  time  to  plan;  planning  completed  in  time  for  execution. 

Execution 

Accurate:  %  of  assets  conforming  to  planed  actions. 

Sufficient:  %  of  assets  completing  assignment. 

Assessment 

Accurate:  %  of  assessments  conforming  to  ground  truth. 

Timely:  time  to  complete  reachback,  personnel  assessment,  ship  threat  assessment. 

Warfighter  Performance 

MDA  Mission 

Mission 

Understanding 

Clear:  1-5  rating  of  understanding  of  activities  and  tasks,  by  position. 

Mission 

Acceptance 

Clear: 

Compliant:  1-5  rating  of  individual,  unit  readiness  to  undertake  MDA  mission,  by  position. 
Compatible:  1-5  rating  on  fit  and  ability  to  perform  MDA  tasks  with  other  duties;  percent  of 
current  tasks  that  match  MDA  needs. 

Unit  Performance 

Manning 

Sufficient:  manning  to  carry  out  assigned  activities,  Y/N,  %  of  required. 

Compatible:  personnel  assigned  with  activity  requirements,  Y/N,  %  match. 

Activities 

Capable:  1-5  rating  of  unit  ability  to  undertake  assigned  MDA  activities. 

Timely:  time  to  complete  activities. 

Compliant:  activity  performance  complies  with  CONOPS,  rules,  agreements,  Y/N. 

Flexible:  1-5  rating  of  unit's  ability  to  respond  to  different  situations. 

Training 

Sufficient:  1-5  rating  of  training  preparation  to  perform  required  activities,  by  activity  and  unit. 
Relevant:  1-5  rating  of  training  relevance  to  assigned  activities,  by  activity  and  unit. 
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Human  Performance 

Tasks 

Capable:  1-5  rating  of  human  ability  to  perform  MDA  tasks,  by  position. 

Timely:  time  to  complete  tasks. 

Flexible:  respond  to  situation. 

Training 

Sufficient:  1-5  rating  of  training  preparation  to  perform  required  activities,  by  task  and  position. 
Relevant:  1-5  rating  of  training  relevance  to  assigned  activities,  by  task  and  position. 

Organization/Guidance 


MDA  Compatibility 

Organization 

Capable:  roll-up  of  fit  of  the  organization's  structure  to  MDA  requirements. 

Alignment 

Sufficient:  rolls  and  decision  making  agreements  to  cover  MDA  contingencies,  Y/N. 

Compatible:  organization  alignment  with  MDA  operations  requirements,  Y/N. 

Flexible:  MDA  responsible  units  can  realign  in  response  to  situation,  Y/N. 

Timely:  organizations  are  self-synchronizing  in  response  to  situation,  Y/N;  time  required  to 
synchronize  operations. 

Usable:  roll-up  of  how  workable  organization  alignment  is  to  accomplish  the  MDA  mission. 

Clear:  responsibilities  and  command  relations  between  MDA  units,  Y/N. 

Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N. 

Manageable:  information  and  decision  flow  between  units,  Y/N. 

Process 

Capable:  roll-up  of  how  capable  MDA  units  are  of  sharing  activities/tasks. 

Alignment 

Sufficient:  information  sharing  agreements  to  cover  MDA  contingencies,  Y/N. 

Compatible:  process  is  in  alignment  with  MDA  operations  requirements,  Y/N;  information 
sharing  between  MDA  units,  Y/N. 

Flexible:  MDA  processes  can  realign  in  response  to  situation,  Y/N. 

Timely:  processes  are  self-synchronizing  in  response  to  situation,  Y/N;  time  required  to 
synchronize  processes. 

Automatic:  information  sharing  between  MDA  units,  Y/N. 

Usable:  roll-up  of  how  workable  activity/task  sharing  is  between  MDA  units. 

Clear:  information  sharing  processes,  Y/N;  activity/task  execution  processes,  Y/N. 

Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N;  information 
provided  by  partner  units,  Y/N. 

Manageable:  information  and  workflow  between  units,  Y/N. 

MHQ/MOC  Compatibility 

Organization 

Capable:  roll-up  of  fit  of  MDA  unit’s  structure  to  MHQ/MOC. 

Alignment 

Sufficient:  agreements  to  cover  MDA  contingencies,  Y/N. 

Compatible:  organization  alignment  with  MDA  operations  requirements,  Y/N. 

Flexible:  MHQ/MOC  and  MDA  units  can  realign  in  response  to  situation,  Y/N. 

Timely:  organizations  are  self-synchronizing  in  response  to  situation,  Y/N;  time  required  to 
synchronize  operations. 

Usable:  roll-up  of  how  workable  alignment  of  MHQ/MOC  with  MDA  units  is  to  accomplish  the  MDA 
mission. 

Clear:  responsibilities  and  command  relations  between  MDA  units,  Y/N. 

Trusted:  partner  units  to  carry  out  their  responsibilities  without  question,  Y/N. 

Manageable:  information  and  decision  flow  between  units,  Y/N. 

Process 

Capable:  roll-up  of  how  capable  MDA  units  are  of  sharing  activities/tasks. 

Alignment 

Sufficient:  information  sharing  agreements  to  cover  MDA  contingencies,  Y/N;  %  of  MDA 
activities  that  map  to  MHQ/MOC  activities. 

Compatible:  processis  alignment  of  MHQ/MOC  and  MDA  operations  requirements,  Y/N; 
information  sharing  between  MHQ/MOC  and  MDA  units,  Y/N;  %  of  activities  that  can  seamlessly 
share  information,  workflow,  and  responsibilities. 

Flexible:  MHQ/MOC  and  MDA  processes  can  realign  in  response  to  situation,  Y/N. 

Timely:  MHQ/MOC  and  MDA  processes  are  self-synchronizing  in  response  to  situation,  Y/N; 
time  required  to  synchronize  processes. 

Automatic:  information  sharing  between  MHQ/MOC  and  MDA  units,  Y/N. 

Usable:  roll-up  of  how  workable  activity/task  sharing  is  between  MHQ/MOC  and  MDA  units. 

Clear:  information  sharing  processes,  Y/N;  activity/task  execution  processes,  Y/N. 

Trusted:  MHQ/MOC  to  carry  out  their  responsibilities  without  question,  Y/N;  information  provided 
by  MHQ/MOC,  Y/N. 

Manageable:  information  and  workflow  between  MHQ/MOC  and  MDA  units,  Y/N. 
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Agreements 

Information 

Sharing 

Reach:  Number  of  different  types  of  information  that  can  be  shared. 

Sufficient:  %  of  required  information  that  can  be  shared. 

Compliant:  with  information  security  regulations,  Y/N;  list  barriers  to  information  sharing  that 
impede  MDA  operations. 

Shared 

Operations 

Clear:  responsibilities,  chain  of  command. 

Sufficient:  %  of  required  units/organizations  participating. 

Reach:  number  of  MDA  activities  that  can  have  shared  participation,  by  activity,  list  activities, 
and  list  participating  units. 

Guidance 

CONOPS 

Usable:  roll-up  assessment  of  CONOPS  quality  to  guide  MDA  operations. 

Sufficient:  guidance  to  conduct  MDA,  by  operation,  Y/N. 

Relevant:  guidance  applies  to  MDA,  by  situation,  Y/N. 

Applicable:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  activities,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives/doctrine,  Y/N. 

Prepare  a  list  of  those  situations  for  which  CONOPS  is  inadequate. 

TTP/SOP 

Usable:  roll-up  assessment  of  TTP/SOP  quality  to  direct  MDA  activities. 

Sufficient:  guidance  to  conduct  MDA,  by  activity,  Y/N. 

Relevant:  guidance  applies  to  MDA,  by  activity,  Y/N. 

Applicable:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  activities,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives,  Y/N. 

Prepare  a  list  of  those  situations  for  which  TTP/SOP  are  inadequate. 

Standing 

Orders 

Usable:  roll-up  assessment  of  Standing  Orders  (ROE,  NSL,  commander's  guidance,  etc.)  quality  to 
frame  MDA  operations. 

Sufficient:  guidance  to  conduct  MDA,  by  operation,  Y/N. 

Relevant:  guidance  applies  to  MDA,  by  situation,  Y/N. 

Applicable:  guidance  can  be  applied,  by  situation,  Y/N. 

Clear:  guidance  to  direct  MDA  operations,  by  activity,  Y/N. 

Compliant:  with  higher-order  directives,  Y/N. 
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Appendix  C:  Attribute  Definitions 

Effective  -  Effective  is  an  overarching  attribute.  It  refers  to  how  well  systems,  people,  and 
processes  meet  their  stated  purposes.  This  attribute  has  meaning  only  in  reference  to  that 
purpose.  E.g.,  it  is  not  sufficient  to  state  that  a  system  is  effective  without  also  stating  at  what. 

Accessible  -  Users  have  access  to  needed  capabilities  and  information.  This  includes  access  to 
communication  means,  data  and  processed  information,  systems,  software,  support,  etc.  Access 
will  often  be  through  a  network.  This  attribute  is  one  of  the  four  MOEs  its  component  MOP 
follows. 

Capacity  -  Number  of  users  that  can  have  access;  number  of  services  that  can  be 
provided;  capacity  of  other  systems  required  for  its  function,  primarily  bandwidth. 
Included  is  information  or  service  throughput. 

Available  -  System  or  capability  is  ready  for  use,  can  be  used,  when  needed.  It  is 
possible  that  a  capability  can  be  accessed  but  cannot  be  used  at  that  time. 

Compatible  -  The  system  or  capability  can  function  with  other  elements  external  to  it 
without  modification  to  either.  It  can  be  integrated  with  other  systems  or  capabilities. 

This  can  also  refer  to  processes  or  organizations  being  compatible  or  integrated. 
Extensive  -  The  system  or  capability  is  capable  of  servicing  a  large  number  of  users, 
covers  a  large  geographical  area,  services  a  large  number  of  user  types,  and  provides  a 
number  of  different  types  of  service. 

Efficient  -  The  number  of  steps  or  effort  needed  to  access  and  use  the  service  is 
acceptable.  This  attribute  is  inherently  comparative.  Acceptable  normally  refers  to  a 
standard,  or  an  improvement  over  what  was  fonnerly  required.  Efficiency  can  be  a  ratio, 
a  judgment  of  (result  obtained)/(effort  required). 

Reliable  -The  capability  or  infonnation  is  there  when  needed,  can  be  depended  on.  Human  and 
organization  reliability  is  included.  This  attribute  is  one  of  the  four  MOE;  its  component  MOP 
follows. 

Robust  -  The  system  or  process  is  able  to  withstand  stress  or  attack.  Changes  in 
environment  are  managed  with  minimal  loss  of  functionality  or  effectiveness. 

Persistent  -  The  system  maintains  its  status  over  long  periods  of  time  (primarily  ISR 
capabilities).  Information  maintains  its  content  and  meaning  across  processing  and 
distribution  means  (e.g.,  tracks). 

Secure  -  The  system,  process,  infonnation,  has  provisions  that  prevent  unauthorized  use, 
intrusion,  or  tampering. 

Assured  -  Information  is  warranted  to  be  correct,  the  source  identified,  and  non¬ 
repudiation  in  effect.  The  process  is  warranted  to  produce  the  desired  result. 

Capable  -  The  system,  capability,  person,  or  organization  provides  the  needed  services.  This 
attribute  is  one  of  the  four  MOE;  its  component  MOP  follows. 

Sufficient  -  What  has  been  provided/received  is  adequate  for  the  recipient  to  perform 
their  function.  For  humans  and  organizations,  the  skills  available  are  adequate  for  task 
performance.  Sufficiency  can  refer  to  either  quantity  or  level. 

Flexible  -  The  system,  process,  human,  or  organization  responds  easily  to  the  situation  or 
to  changing  requirements.  It  is  adaptable,  can  handle/utilize  a  wide  range  of  types.  It  is 
tailorable/customizable  to  user  needs  and/or  users  can  make  modifications  to  suite  their 
needs. 
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Accurate  -  Information  provided  is  correct,  matches  reality  within  acceptable  limits. 
Determinations  of  accuracy  normally  require  definition  of  acceptable  error  limits. 

Timely  -  The  occurrence  or  delivery  is  within  acceptable  time  limits.  This  can  refer  to 
an  elapsed  time  or  to  meeting  a  schedule. 

Usable  -  The  system,  capability,  information,  or  process  can  be  used.  This  attribute  is  one  of  the 
four  MOE;  its  component  MOP  follows. 

Clear  -  How  the  system  or  process  is  to  be  used  is  easily  understood.  Meaning  of  the 
information  is  easily  comprehended.  Instructions,  guidelines,  definitions  are  complete 
and  meaningful. 

Trusted  -  Users  believe  that  the  infonnation,  process,  system,  organization,  will  perform 
their  function  in  a  manner  that  supports  current  needs. 

Manageable  -  The  system  or  process  can  be  easily  modified  or  manipulated  as  needs 
dictate,  often  in  response  to  changes  in  the  environment.  Included  is  insuring  that  the 
required  level  of  performance  is  maintained.  This  includes  installation  of  capabilities. 
Relevant  -  Information  provided  applies  to  the  current  situation.  System  capabilities  are 
what  is  needed  for  current  tasks.  Processes  provide  the  actions  required  for  current 
operations. 

Compliant  -  The  system  or  infonnation  complies  with  standards  or  defined  structure  and 
formats.  Activities  are  in  conformance  with  existing  CONOPS  and  TTP. 

Military  Utility  -  Military  utility  the  second  overarching  attribute,  and  actually  is  a  faux  attribute, 
not  actually  a  description  of  characteristics  but  a  determination  to  be  made  in  Military  Utility 
Assessments  (MU A).  It  is  used  to  express  that  something  does/does  not  contribute  to  the 
successful  performance  of  military  operations.  It  is  one  of  the  most  important  considerations  for 
military  operations.  The  four  measures  of  utility  (MOU)  follow. 

Improved  -  The  system,  organization,  or  process  improves  the  conduct  of  military 
operations  for  which  they  were  designed. 

Needed  -  The  system,  organization,  or  process  fills  a  gap  an  identified  gap. 

Applicable  -  The  system,  organization,  or  process  is  pertinent  to  conduct  of  the 
operation.  Its  capabilities  match  the  needs  and  conduct  of  the  operation. 

Wanted  -  Operational  personnel  want  the  capability  and  utilize  it.  They  do  not  currently 
have  the  capability  or  would  rather  use  it  in  place  of  other  available  capabilities. 

Ready  Ready  is  an  official  procurement  term  that  refers  to  the  system  being  ready  for  fielding. 
As  indicated,  it  is  a  roll-up  of  the  other  fundamental  measures  and  the  life-cycle  plan  (which 
includes  a  personnel  plan). 
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Appendix  D:  Potential  Deficiencies  regarding  System  Performance 


Sub-Sub-Area 

Technology 

Item 

System  Performance: 
System  Management 
and  Security 

CMA 

CMA:  Users  sometimes  lost  track  of  the  original  security 
classification  of  the  information  they  wished  to  disseminate.  This 
increased  the  likelihood  of  a  security  violation  as  a  result  of  passing 
classified  information  on  the  wrong  domain. 

System  Performance: 
Technical  Performance: 
Information  Sharing 

Google  Earth 

Google  Earth  has  no  embedded  collaboration  tool  included,  thus  it 
was  not  possible  to  send  information  from  Google  Earth  to  other 
systems. 

System  Performance: 
Automation:  Alerts 

All  Spiral- 1 

Spiral- 1  did  not  automatically  establish  or  display  threat  assignments 
based  upon  a  user-defined  alert. 

System  Performance: 
Automation:  Alerts 

CMA  & 
FASTC2AP 

Spiral- 1  tools  did  not  provide  additional  capability  to  establish 
baseline  normal  civil  maritime  operations  worldwide  and  threat 
assessment  criteria.  CMA  and  FASTC2AP  could  alert  based  upon  a 
geographic  point/area/proximity,  but  did  not  support  alerts 
employing  algorithms  based  upon  baseline  maritime  operations. 

System  Performance: 
Operations  Support: 
System  Utility 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  Although  wireless,  the  TED  devices 
were  required  to  be  in  the  vicinity  of  the  Tactical  EMIO  Maritime  PC 
(TEMP)  in  order  to  download  data  captured  during  the  boarding.  The 
radio  frequency  (RF)  signals  were  not  strong  enough  to  transmit  data 
when  team  members  were  below  decks.  (Visit,  Board,  Search  and 
Seizure  (VBSS)  School) 

System  Performance: 
Operations  Support: 
System  Utility 

E-MIO 

Wireless 

Tactical  EMIO  System  (TES)  -  While  mobility  of  the  Tactical  EMIO 
Device  (TED)  was  a  clear  advantage,  enabling  the  collection  of  data 
from  multiple  locations  within  the  vessel,  one  limitation  noted  was 
that  the  TED  must  be  within  the  vicinity  of  the  TEMP  to  download 
the  data  captured.  The  RF  signals  were  not  strong  enough  to 
transmit  data  when  team  members  were  below  decks.  The  Maritime 
BGAN  EMIO  Terminal  (MBET)  device,  in  turn,  failed  to  transfer 
data  due  to  environmental  issues  and  weak  RF  signal  range  of  the 
commercial  satellite.  Contractors  eventually  departed  the  target 
vessel  and  drove  inland  with  the  TEMP  and  MBET  device  to  acquire 
a  stronger  signal.  The  MBET  link  was  then  acquired  and  successfully 
transmitted  data  from  the  TEMP  device.  Contractors  asserted  that  the 
satellite  connectivity  will  not  be  a  concern  in  the  current  AOR.  Also, 
the  boarding  officer  was  not  able  to  demonstrate  the  transfer  of  data 
via  the  Maritime  Broadband  Global  Area  Network  (BGAN)  EMIO 
Terminal  (MBET)  due  to  the  satellite  connectivity.  (VBSS  School) 

System  Performance: 
Automation:  Alerts 

All  Spiral- 1 

The  ability  to  capture  and  store  baseline/normal  maritime  movement 
patterns  was  not  observed.  Spiral- 1  tools  did  not  alert  users  to 
deviations  from  normal  route  or  behavior  patterns. 

System  Performance: 
Technical 

Performance: 
Information  Retrieval 

CMA 

The  user’s  ability  to  monitor  vessel,  person,  and  cargo  data  was 
severely  degraded  by  gaps  in  track  data  coverage.  When  a  node’s 
CMA  server  was  down,  or  data  was  not  transmitted,  the  data  not 
received  was  not  recoverable.  The  Naval  Research  Laboratory  limited 
the  National  Technical  Means  data  source  input  to  CMA  to  14  hours 
per  day  and  filtered  the  data  that  was  provided.  This  resulted  in  a  gap 
of  data  which  had  a  negative  impact  across  all  AORs. 

System  Performance: 
Technical  Performance 
Information  Sharing 

All  Spiral- 1 

There  were  significant  differences  in  information  available  at 
different  nodes. 
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Appendix  E:  Potential  Deficiencies  regarding  Operations  Performance 


Sub-Sub- Area 

Technology 

Item 

Operations  Perfonnance: 
Knowledge  Processes: 

Vol  Development  & 
Operations  Perfonnance: 
MIO:  Execution 

E-MIO 

Wireless 

TES  -  While  mobility  of  the  Tactical  EMIO  Device 
(TED)  was  a  clear  advantage,  enabling  the  collection 
of  data  from  multiple  locations  within  the  vessel,  one 
limitation  noted  was  that  the  TED  must  be  within  the 
vicinity  of  the  TEMP  to  download  the  data  captured. 
Although  wireless,  the  TED  devices  were  required  to 
be  in  the  vicinity  of  the  TEMP  in  order  to  download 
data  captured  during  the  boarding.  The  radio 
frequency  (RF)  signals  were  not  strong  enough  to 
transmit  data  when  team  members  were  below  decks. 
The  MBET  device,  in  turn,  failed  to  transfer  data  due 
to  environmental  issues  and  weak  RF  Signal  range  of 
the  commercial  satellite.  Contractors  eventually 
departed  the  target  vessel  and  drove  inland  with  the 
TEMP  and  MBET  device  to  acquire  a  stronger  signal. 
The  MBET  link  was  then  acquired  and  successfully 
transmitted  data  from  the  TEMP  device.  Contractors 
asserted  that  the  satellite  connectivity  will  not  be  a 
concern  in  the  current  AOR.  Also,  the  boarding 
officer  was  not  able  to  demonstrate  the  transfer  of 
data  via  the  MBET  due  to  the  satellite  connectivity. 
(VBSS  School) 

Operations  Perfonnance: 
Knowledge  Processes: 

Vol  Tracking 

CMA 

The  user’s  ability  to  monitor  vessel,  person,  and 
cargo  data  was  severely  degraded  by  gaps  in  track 
data  coverage.  When  a  node’s  CMA  server  was 
down,  or  data  was  not  transmitted,  the  data  not 
received  was  not  recoverable.  The  Naval  Research 
Laboratory  limited  the  National  Technical  Means 
data  source  input  to  CMA  to  14  hours  per  day  and 
filtered  the  data  that  was  provided.  This  resulted  in  a 
gap  of  data  which  had  a  negative  impact  across  all 
areas  of  responsibility  (AORs). 

Operations  Perfonnance: 
MIO:  Execution 

E-MIO 

Wireless 

TES  -  Although  wireless,  the  TED  devices  were 
required  to  be  in  the  vicinity  of  the  TEMP  in  order  to 
download  data  captured  during  the  boarding.  The 
radio  frequency  (RF)  signals  were  not  storing  enough 
to  transmit  data  when  team  members  were  below 
decks.  (VBSS  School) 

Operations  Perfonnance: 
MIO:  Execution 

E-MIO 

There  were  no  positions  provided  with  the  latent  print 
messages  and  for  the  cave  collection,  there  was  no 

Seek  ID  Global  Positioning  System  (GPS)  reception 
and  therefore  no  position  was  included  in  the 
biometric  messages. 
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Appendix  F:  Average  Assessment  Scores  for  MDA  Technologies 


Sub-Sub-Area 

CMA 

E-MIO 

Wireless 

FASTC2AP 

Global 

Trader 

Google  Earth 

LiNX 

MAGNET 

MASTER 

MDA  DS 

COI 

MIDAS 

PANDA 

Tripwire 

All  MDA 

Tech 

Average 

Operations  Performance:  Knowledge  Processes:  Vol  Development 

2.5 

1.0 

2.0 

3.0 

2.1 

2.0 

3.0 

3.0 

3.0 

2.5 

2.5 

Operations  Performance:  Knowledge  Processes:  Vol  Tracking 

2.3 

3.0 

2.4 

Operations  Performance:  MIO:  Execution 

1.8 

1.8 

Organization/Guidance:  Agreements:  Information  Sharing 

2.0 

2.0 

2.0 

Organization/Guidance:  Guidance:  Concept  of  Operations  (CONOPS) 

2.0 

2.0 

Organization/Guidance:  Guidance:  Tactics,  Techniques,  Procedures 
(TTP/SOP) 

2.0 

2.0 

Organization/Guidance:  MDA  Compatibility:  Organization  Alignment 

2.0 

2.0 

Organization/Guidance:  MHQ/MOC  Compatibility:  Organization 
Alignment 

2.0 

2.0 

Organization/Guidance:  MHQ/MOC  Compatibility:  Process  Alignment 

3.0 

3.0 

System  Performance:  System  Management  and  Security 

1.7 

2.5 

2.0 

System  Performance:  Automation:  Alerts 

2.3 

2.3 

2.5 

3.0 

3.0 

1.3 

2.3 

System  Performance:  Automation:  Information  Processing 

3.0 

3.0 

System  Performance:  Operations  Support:  Standards  and  Guidelines 

2.0 

2.0 

System  Performance:  Operations  Support:  System  Utility 

3.0 

2.0 

3.0 

3.0 

3.0 

2.0 

3.0 

3.0 

2.3 

2.6 

System  Performance:  Technical  Performance:  Information  Processing 

2.0 

2.0 

System  Performance:  Technical  Performance:  Information  Retrieval 

2.3 

2.0 

3.0 

2.5 

2.2 

3.0 

2.5 

2.0 

2.3 

2.3 

System  Performance:  Technical  Performance:  Info.  Sharing 

2.2 

3.0 

2.0 

1.5 

3.0 

2.0 

2.0 

3.0 

1.8 

2.2 

System  Performance:  Technical  Performance:  Interoperability 

2.0 

2.0 

2.0 

2.0 

System  Performance:  Warfighter  Acceptance:  Human-System 

Interaction 

2.5 

2.0 

2.3 

3.0 

2.0 

2.0 

3.0 

2.0 

2.4 

System  Performance:  Warfighter  Acceptance:  System  Training 

2.5 

2.4 

2.0 

2.0 

2.0 

2.3 

System  Performance:  Warfighter  Acceptance:  System  Usage 

3.0 

3.0 

2.5 

2.8 

System  Performance:  Warfighter  Acceptance:  System  Utility 

2.8 

2.0 

2.5 

2.0 

2.5 

2.0 

3.0 

3.0 

2.6 

System  Supportability  and  Readiness:  System  Supportability  and 
Readiness 

2.0 

2.0 

Average 

2.4 

2.1 

2.4 

2.2 

2.5 

2.4 

2.0 

2.3 

2.0 

2.6 

2.9 

2.9 

2.1 

2.4 
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Appendix  G:  Mapping  of  SP-1  Technologies  to  MDA  Capabilities  Thresholds 


This  table  presents  a  fine-grained  mapping  of  Spiral- 1  technologies  to  MDA  capabilities. 


Capability 

Class 

Capability 

CMA 

E-MIO 

Wireless 

Global 

Trader 

Google 

Earth 

LiNX 

MAGNET 

Data  Sharing 
COI 

Tripwire 

FASTC2AP 

Potential 

Gaps 

Potential 

Redundancy 

Monitor 

MT1 

X 

X 

X 

X 

X 

Y 

MT2 

X 

X 

X 

X 

Y 

MT3 

X 

Y 

MT4 

X 

X 

X 

X 

Y 

MT5 

X 

Y 

Collect 

CT1 

X 

X 

X 

X 

X 

X 

Y 

CT2 

X 

Y 

CT3 

Y 

CT4 

X 

Y 

CT5 

X 

X 

Y 

Fuse 

FT1 

X 

X 

X 

Y 

FT2 

X 

Y 

Analyze 

ATI 

X 

X 

Y 

AT2 

Y 

AT3 

X 

X 

Y 

AT4 

X 

Y 

AT5 

Y 

AT6 

X 

Y 

Disseminate 

DTI 

X 

Y 

DT2 

Y 

DT3 

X 

X 

X 

Y 

DT4 

X 

Y 

DT5 

X 

X 

Y 

DT6 

X 

Y 

DT7 

X 

Y 

DT8 

X 

X 

X 

Y 

DT9 

X 

Y 

Note:  This  Table  was  developed  from  a  presentation  by  PEO  C4I  concerning  MDA  test  and  evaluation,  21  Feb  2008  (PEO_C4I_MDA_TE  Update  Issues  FINAL  rev2_20FEB2008.ppt) 
Note:  MDA  Capability  Thresholds  are  drawn  from  the  OPNAV  Scoping  Document 
Note:  Key  to  capabilities. 
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Appendix  H:  Organization/Guidance  Results 


Item 

ID 

Area 

Item 

Comments 

Score 

Technol¬ 

ogy 

Venue 

295 

Agreements: 

Information 

Sharing 

Information  from  a  stand-alone  MAGNET  system  was  not  available  due  to 
sharing  agreements  concerning  U.S.  persons  information. 

constraints  imposed  by 
policies  and  agreements 

2 

MAGNET 

QRA 

296 

Agreements: 

Information 

Sharing 

MIDAS:  Respondents  acknowledged  that  there  is  currently  a  need  to 
collaborate  with  other  agencies,  coalition  members  and  non-traditional 
partners,  and  most  felt  that  MIDAS  would  help  to  facilitate  that  collaboration; 
however,  2  mentioned  that  sharing  sensitive  or  classified  information  might  be 
a  problem. 

constraints  imposed  by 
policies  and  agreements 

2 

MIDAS 

TW08 

299 

Guidance: 

CONOPS 

CMA  -  The  envisioned  CONOPS  for  future  MDA  will  require  training  and 
familiarization  with  the  new  procedures  as  evidenced  by  the  following:  (1) 
"Have  yet  to  truly  collaborate  with  others,  with  the  exception  of  perhaps  shared 
watch  areas;"  and  (2)  "During  the  exercise  there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come  near  a  specific  vessel  at  any  time  during 
its  track  - 1  still  don’t  think  this  function  is  truly  possible." 

training  for  CONOPS 

2 

CMA 

VBSS 

302 

Guidance: 

TTP/SOP 

CMA  -  The  envisioned  TTP/SOP  for  future  MDA  will  require  training  and 
familiarization  with  the  new  procedures  as  evidenced  by  the  following:  (1) 
"Have  yet  to  truly  collaborate  with  others,  with  the  exception  of  perhaps  shared 
watch  areas;"  and  (2)  "During  the  exercise  there  was  a  lot  of  confusion  about 
being  able  to  track  vessels  that  come  near  a  specific  vessel  at  any  time  during 
its  track  - 1  still  don’t  think  this  function  is  truly  possible." 

training  for  TTP 

2 

CMA 

VBSS 

290 

MDA  Compatibility: 

Organization 

Alignment 

ONI  expresses  concern  about  lack  of  lack  of  re-engineering  of  processes  and 
training. 

coordination  of  MDA 
and  ONI  processes 

2 

All  MDA 
Tech 

TW08 

304 

MDA  Compatibility: 

Organization 

Alignment 

ONI  is  conducting  process  analyses  concerning  intel  analysis.  It  is  not  clear 
that  this  effort  is  synchronized  with  the  MDA  technology  effort. 

coordination  of  MDA 
with  ONI  processes 

2 

All  MDA 
Tech 

Workflow, 

PEW 

276 

MHQ/MOC 

Compatibility: 

Organization 

Alignment 

The  NPS  assessment  team  is  concerned  that  MDA  TTPs  need  to  be  sufficiently 
flexible  to  accommodate  differences  between  COCOMS. 

customization  to  local 
needs 

2 

All  MDA 
Tech 

TW08 

303 

MHQ/MOC  Compat¬ 
ibility:  Process 
Alignment 

Second  Fleet,  NAVNETWARCOM,  and  others  concur  that  the  MDA 
workflow  aligns  with  the  MHQ  w  MOC  process  architecture  as  of  early  2008. 
(See  Process  Alignment  Workshop). 

alignment  of  MDA 
and  MHQ  w  MOC 
processes 

3 

All  MDA 
Tech 

TW08 
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14th  ICCRTS:  C2  and  Agility 


1 


NAVAL 

POSTGRADUATE 

SCHOOL 


MDA  Spiral  1  Tools,  Features  Provided  and  Benefits  of  Tools 


Tool  Features  Provided 


Benefits  of  Tool 


Comprehensive  Maritime  Awareness  (CMA) 

Anomaly  detection  *  Automates  a  previously  manual  process 

Info  sharing  across  organizational  nodes  *  Facilitates  collaboration 

Object  tracking  *  More  tracks  can  be  monitored 

Integrates  multi-source  data,  search 
agents,  confidence  reporting,  and 
information 

remote  access 


*  More  comprehensive  correlation 

across  multiple  sources  of 


FASTC2AP 


TRIPWIRE 


analyzed 


Defines/automates  info  mgmt  tasks 
Helps  manage  situation  awareness 
Processes  large  amounts  of  data 
by  analysis,  fusion,  and  exploitation 
agents 

Mines  unstructured  text  data  and 
alerts  analysts 


*  Generates  alerts,  emails,  OTH  Gold 

msgs  -  update  GCCS-M  tracks 

*  Access  to  multiple  databases 

*  Helps  manage  situation  awareness  and 
track  behavior  in  an  AOR 

*  Increased  relationships 


MDA  Spiral  1  Tools,  Features  Provided  and  Benefits  of  Tools 


Tool  Features  Provided 

Benefits  of  Tool 

MAGNET 

Intelligent  agent  logic 

Web  browser  to  query  other  data 

time 

*  Faster  access  to  data 

*  More  sources  queried  in  less 

sources 

System  queries  regarding  topics  of 

analysis 

interest  in  area 

*  Increased  number  of  analytic  models 
*  Increased  archived  data  for 

TAANDEM 

Real-time  track  processing 

Identify  deviant  tracks 

*  Prediction  and  activity  monitoring 
*  Real-time  track  processing 

Provides  annotated  tracks  with  alerts  *  Rule-based  anomaly  detection 

E-MIO  Wireless 

Satellite  transmission/  reception  for  *  Automatically  inserts  data  into 

unclassified  boarding  data  authoritative  dbs 


GOOGLE 

EARTH  Map  capabilities 

*  Overlay  analysis  from  other  tools 
to  detailed  map 

Global  Trader 

Anomaly  detection,  pattern  matching/ 

*  Supports  queries  regarding  cargo 

Maritime  Domain  Awareness:  Spiral  1  Metrics 


Program  Metrics  Structure 


Structure  is  required  to  support  results  roll-up  to  higher-level  metrics. 


Effective 


Accessible 


Reliable 


Capable 


Usable 


MOE 


Capacity 

Available 

Compatible 

Extensive 

Efficient 


Robust 

Persistent 

Secure 

Assured 


H — 


Sufficient 

Flexible 

Accurate 

Timely 

Reach 

Automatic 


Clear 

Trusted 

Manageable 

Relevant 

Compliant 

Deployable 


MOP 


Military  Utility 
Improved 


Needed 


Applicable 


Wanted 


MOU 


Readiness  in  the  acquisition  sense,  not  operational  or  deployment  readiness. 


4 


rtRKit-Mt-m 


Assessment  Framework  Addresses  Five  Distinct 

Assessment  Areas 

•  System  Performance  concerns  how  well  a  system  performs  its  functions,  its 

support  of  MDA  operations,  warfighter  acceptance,  automation,  and  system 
management  and  security  functions. 

•  Operations  Performance  addresses  the  quality  of  knowledge  management 
concerning  vessels  of  interest  (Vols),  MDA  intelligence,  surveillance,  and 
reconnaissance  (ISR),  and  MIO. 

•  Warfighter  Performance  focuses  on  operator  acceptance  and 
understanding  of  the  MDA  mission,  as  well  as  unit  and  individual  capability 
to  execute  that  mission. 

•  Organization/Guidance  focuses  on  the  fit  of  organizational  structures  and 
processes  to  the  MDA  mission  including  Maritime  Headquarters  with 
Maritime  Operations  Center  (MOC),  the  sufficiency  of  agreements  between 
entities,  and  the  adequacy  of  guidance  within  entities. 

•  Supportability  and  Readiness  concerns  factors  that  ensure  MDA  Spiral-1 
systems  are  robust  and  reliable. 


MDA  Capability  Indicators  (or  Assessment  Metrics) 


Area 

Effective 

Utility 

Area 

Effective 

System 

Accessible 

Improved 

Warfighter 

Performance 

Capable 

Needed 

Performance 

Capable 

Reliable 

Usable 

Applicable 

Wanted 

Reliable 

Operations 

Accessible 

Organization 

Accessible 

Capable 

/Guidance 

Capable 

Reliable 

Usable 

6 


Example  of  MDA  Assessment  Metrics:  Assessment  Area, 

Capability  Indicators,  and  Metrics 


System  Performance 
Assessment  Area 

Capability 

Indicators 

Metrics 

Technical  Performance 

Improved 

-5  to  +5  rating  of  improvement  over  existing  system 
by  system  feature 

Needed 

System  fills  a  gap  in  existing  capabilities,  Y/N 

Applicable 

System  is  applicable  to  MDA  activities,  by  activity, 

Y/N 

Wanted 

-5  to  +5  rating  of  operator  desire  to  have  system 
available 

Information  Retrieval 

Accessible 

Roll-up  of  information  accessibility 

Available 

%  of  time  information  is  available 

Efficient 

Number  of  steps  to  access  information 

Capable 

Roll-up  of  capability  to  retrieve  required  information 

Sufficient 

%  of  information  needed  for  assessment 

Timely 

Time  required  to  retrieve  information 

Data  Collection 


ymsiANiu  n*  kiistum 


1.  Workflow  Analysis  and  Process  Engineering  Workshop  (PEW): 

>  Overview  of  MDA  activities,  site  visits  ton  NAVCENT,  ONI,  2F,  3F,  5F,  and  6F. 

>  Document  the  way  MDA  is  currently  done  and  to  identify  where  new  tech’s 
are  expected  to  have  the  greatest  impact. 

>  Validated  in  PEW,  NPS,  Jan  2008,  and  was  mapped  to  the  Maritime  Headqrtrs 

with  Maritime  Operations  Center  (MHQ  w/MOC)  core  processes. 

2.  Technical  Risk  Reduction  Limited  Objective  Experiment,  SPAWAR,  SD,  June  2008 

>  First  simultaneous  testing  of  multiple  MDA  technologies  using  Fleet 
participants. 

3.  Visit  Board  Search  and  Seizure  (VBSS)  School,  San  Diego 

4.  Trident  Warrior  08  -  held  aboard  multiple  Navy  ships 

5.  Empire  Challenge  08  -  China  Lake  Naval  Station  and  in  San  Francisco  Harbor,  2008. 

6.  FAIRGAME  -  Simultaneous,  multiple  sites:  Naval  Maritime  Intelligence 

Center  (NMIC),  NAVCENT,  PACFLT,  Maritime  Intelligence  Fusion  Center,  Pacific 
(MIFCPAC),  MIFCLANT,  SSC-SD  and  NCIS,  July  2008 

>  Primary  source  for  COMOPTEVFOR  to  perform  the  QRA  on  all  MDA  technologies 


MARLO:  Intel: 
Generate  tip  intel 
Wants:  FastC2AP;  CEN 


CIFC:  Intel: 
Generate  tip  intel 
Wants:  FastC2AP:  CENTRIX 

i  F;  P;  E;  C;  B 


Observe  &  Orient 


NCIS:  Intel: 
Generate  tip  intel 
Wants:  FastC2AP:  CENTRIX 


ONI:  Intel: 
Generate  tip  intel 
Wants:  FastC2AP:  CEN 


Inti  Maritime  Bureau:  lnteh\50) 
Generate  tip  intel 
Wants:  FastC2AP:  CENTRIX 


ONA:  VOI: 

Specify  VOI  &  credibility  of  intel 

(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  SMSJPSC2;  CENTRIX 


Decide 


MOC  Director:  Receive  VOI: 
Receive  &  route  VOI 
Wants:  CENTRIX 


KEY 

B 

Briefs 

C 

Chat 

E 

Email 

F 

Face  to  Face 

M 

MsgTraffic 

P 

Phone 

t 

To  Be  Determined 

| - 1 

MOC 

o 

External  Entity 

— ► 

Method  of  Comm.  Not 

Specified 

1  1 

No  Arrow  Going  Out 

COPS  Director  or  BWC:  Process  VOI: 
Process  VOI  (20%  of  VOIs) 
Wants:  CENTRIX 


Act 


F;  e 


FOPS:  Process  VOI: 

Process  VOI  (80%  of  VOIs),  Issue  RFI 
Wants:  CENTRIX 


F;  E 


BWC:  Assess  tactical  assets 
Assess  tactical  asset  availability 
(Currently  use  SeaLink  for  historical  positioning  data) 
_ Wants:  FastC2AP;  SMSJPSC2;  CENTRIX 


MOC  Director:  Define  COA: 
Define  COA,  CAT1-4 
Wants:  CENTRIX 


MOC:  MOC-MOC  handoff: 
Hand  off  mission  to 
another  MOC 

(see  attached  workflow) 

Wants:  TBD 


IWO:  Process  or  Issue  RFI: 
Issue  RFI 

Wants:  FastC2AP;  CENTRIX 


ONA:  Process  RFI: 

Process  RFI 

(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  CENTRIX 


ONI:  Process  RFI 
Process  RFI 
Wants:  FastC2AP;  CEN1 


NCIS,  CIFC,  MARLO,  NGA:  Process  RF 
Process  RFI  (May  use  SeaLink) 
Wants:  FastC2AP;  CENTRIX 


5th  Fleet:  Execute  VBSS  mission: 

Execute  VBSS  mission  (see  attached  workflow) 
(Currently  use  SeaLink  for  historical  positioning  data) 
Wants:  CENTRIX 

M. 


E;  M;  C 


ISR:  Data  Collection 


F;  P;  E;  C 


BWC:  Comm  orders: 
Communicate 
mission  orders 
Wants:  CENTRIX 


CIFC:  Execute  VBSS  mission: 

Execute  VBSS  mission 
Wants:  FastC2AP;  CENTRIX 


E;  M;  C 


COPS:  Monitor  VBSS: 
Monitor  VBSS  Execution 
Wants:  FastC2AP;  CENTRIX 


E;  M;  C 


5th  Fleet:  Take  biometricsTi^g® 
Take  biometric  data 
Wants:  E-MIO 


MOC:  Forward  biometrics:: 
Relay  biometrics  data 
Wants: 


ONI:  ^ 

Analysis/Report 

1 - 


BFC  (WV):Analyze  biometrics: 
Analyze  fingerprints.  (Hits  go  to  ONI  &  0 
Wants: 


E 


MOC:  Change  mission: 
Recommend  revision  of 
CAT  level  per  findings 
Wants: 


MOC:  Complete  mission: ' 
Mission  completed 
(success  or  failure) 
Wants: 
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F;  E 


ONA:  Monitor  VOI:  \ 

Monitor  VOI  (watch  list)  for  6  months 
(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  CENTRIX 


F;  E 


ONI:  Analyze  boarding  data: 
Analyze  imagery,  vessel  parameters 
Wants: 


I 


5th  Fleet:  Take  boarding  d 
Take  boarding  data 
Wants: 


ONA:  Analyze  findings:  v 

Analyze  findings  from  ONI.  Advise  COPS 
(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  CENTRIX 


Table  3.  Distribution  of  Data  by  Assessment  Area 


Area 

Number  Data  Items 

Average  Score 

Operations  Performance 

70 

2.4 

Organization/Guidance 

8 

2.1 

System  Performance 

219 

2.4 

System  Supportability  and  Readiness 

7 

2.0 

Warfighter  Performance 

0 

- 

Total  /  Average 

304 

2.4 

Table  4.  Distribution  of  Data  by  Score  and  Assessment  Area 


Score 

Assessment  Area 

1:  Deficiency 

2:  Concern 

3:  Strength 

Total 

Operations  Performance 

5 

33 

32 

70 

Organization/Guidance 

7 

1 

8 

System  Performance 

10 

119 

90 

219 

System  Supportability  and 
Readiness 

- 

7 

- 

7 

Total 

15 

166 

123 

1304 

Distribution  of  Data  by  Technology 


Technology 

Frequency 

Average  Score 

All  Spiral-1 

47 

2.1 

CMA 

136 

2.4 

E-MIO  Wireless 

23 

2.1 

FASTC2AP 

13 

2.4 

Global  Trader 

5 

2.2 

Google  Earth 

10 

2.5 

LiNX 

5 

2.4 

MAGNET 

2 

2.0 

MASTER 

33 

2.3 

MDA  DS  COI 

5 

2.0 

MIDAS 

5 

2.6 

PANDA 

13 

2.9 

Tripwire 

7 

2.9 

Total/  Avg 

304 

2.4 

ii 

Average  Sum  of  Concern  and  Strength  Scores  by 

Technology 


MDA  Technology 

Avg  Sum  of  Scores 

FASTC2AP 

3.0 

Global  Trader 

3.0 

PANDA 

3.0 

Tripwire 

3.0 

CMA 

2.8 

MASTER 

2.7 

Google  Earth 

2.5 

LiNX 

2.3 

MDA  DS  COI 

2.0 

E-MIO  Wireless 

1.5 

MAGNET 

n.a. 
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Mapping  of  SP-1  Technologies  to  MDA  Capabilities 


Capability 

CMA 

E-MIO 

Wireless 

Global 

Trader 

Google 

Earth 

LiNX 

MAG 

-NET 

Data 

Sharing 

COI 

Trip¬ 

wire 

FAST 

C2AP 

Assessment 

Score 

2.4 

2.1 

2.2 

2.5 

2.4 

2.0 

2.0 

2.9 

2.4 

Monitor 

X 

X 

X 

X 

X 

X 

Collect 

X 

X 

X 

X 

X 

X 

X 

X 

Fuse 

X 

X 

X 

X 

Analyze 

X 

X 

X 

X 

X 

Disseminate 

X 

X 

X 

X 

X 

X 

X 

13 


Summary  of  Findings:  Areas  of  Greatest  Concern 

^  Operations  Performance  for  EMIO 

>  Several  sub-areas  of  Organization/Guidance  concerning 
MDA  Compatibility  (specifically,  alignment  of  MDA 
activity  with  ONI) 

>  Guidance  (especially  the  need  for  MDA  CONOPS,  TTPs, 
and  SOPs) 

>  Agreements  (concerning  data  sharing) 

>  System  Management 

>  Security 

>  System  Supportability 

>  Readiness 
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Mapping  of  SP-1  Technologies  to  MDA  Capabilities  Thresholds 


Capability  Capabil-  CMA  E-MIO  Global  Google  LiNX  MAG 
Class  ity  Wireless  Trader  Earth  -NET 


Monitor 


KEY 

B 

Briefs 

C 

Chat 

E 

Email 

F 

Face  to  Face 

M 

MsgTraffic 

P 

Phone 

t 

To  Be  Determined 

MOC 

External  Entity 

r  /C: 

r 

1 

r 

ONA:  Monitor  VOI:  ^ 

Monitor  VOI  (watch  list)  for  6  months 
(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  CENTRIX 

1 

ONA:  Analyze  findings: 

Analyze  findings  from  ONI.  Advise  COP^_0 
(Currently  use  SeaLink,  Intellipedia,  Centrix,  GoogleEarth,  CMA) 
Wants:  CMA;  MAGNET;  FastC2AP;  GoogleEarth;  CENTRIX 

